Streaming/Windowed GeoTiff Reading by echeipesh · Pull Request #1905 · locationtech/geotrellis

echeipesh · 2016-12-12T13:42:18Z

The main culprit here is StreamingSegmentBytes. In order to do its job it needs to know ahead of time which segments you will read. At that point it can use the TiffTags to group them into contiguous chunks which can be fetched in on-disk order. This requires slight inversion of control between segmentBytes and GeoTiffTile where we allow the bytes to decide the order in which we see the segment. Overall it looks like this:

cfor(0)(_ < segmentCount, _ + 1) { segmentIndex =>
   val segment = getSegment(segmentIndex).bytes
   val segmentSize = segment.size
   val bandSegmentCount = segmentSize / bandCount
   val bandSegment = Array.ofDim[Byte](bandSegmentCount)
...

is replaced by

getSegments(0 until segmentCount).foreach { case (segmentIndex, geoTiffSegment) =>
    val segment = geoTiffSegment.bytes
    val segmentSize = segment.size
    val bandSegmentCount = segmentSize / bandCount
    val bandSegment = Array.ofDim[Byte](bandSegmentCount)
...

Because getSegments gives us an iterator we can zip them and use them for combine as well:

getSegments(0 until segmentCount)
  .zip(otherGeoTiff.getSegments(0 until segmentCount))
  .foreach { case ((segmentIndex, segment), (otherIndex, otherSegment)) =>
    require(segmentIndex == otherIndex, s"Segment index mismatch: $segmentIndex != $otherIndex")
    val newBytes = segment.mapWithIndex { (i, z) =>
      f(z, otherSegment.getInt(i))
    }
    arr(segmentIndex) = compressor.compress(newBytes, segmentIndex)
  }

Things that need done before this is done:

Test on S3 and verify performance on reading fully and windowed
Test on S3 file reading multiple crops from the same GeoTiff (verify that byteReader doesn't Reset)
Replace all usages of getSegment with getSegments in GeoTiffMultiBandTile (some a tricksy)
Update the StreamingSegmentBytesSpec for current code (restored LazySegmentBytes)

Signed-off-by: jbouffard <[email protected]>

…entBytes Signed-off-by: jbouffard <[email protected]>

…SegmentBytes Signed-off-by: jbouffard <[email protected]>

Signed-off-by: jbouffard <[email protected]>

…made in StreamingSegmentBytes Signed-off-by: jbouffard <[email protected]>

…streaming Signed-off-by: Grigory Pomadchin <[email protected]>

Signed-off-by: Grigory Pomadchin <[email protected]>

…s function Signed-off-by: Grigory Pomadchin <[email protected]>

Signed-off-by: Grigory Pomadchin <[email protected]>

pomadchin · 2017-03-02T19:47:53Z

Took into account all comments; StreamingSegmentBytesSpec is still needs to be done.
@echeipesh @lossyrob

Signed-off-by: Grigory Pomadchin <[email protected]>

lossyrob · 2017-03-07T01:58:09Z

  }

-  private def readGeoTiffInfo(byteReader: ByteReader, decompress: Boolean, streaming: Boolean): GeoTiffInfo = {
+  private def readGeoTiffInfo(byteReader: ByteReader, decompress: Boolean, streaming: Boolean, extent: Option[Extent]): GeoTiffInfo = {


Where is the Option[Extent] used? Seems unused.

lossyrob · 2017-03-07T02:00:22Z

+                      ) extends SegmentBytes with LazyLogging {
+  import LazySegmentBytes.Segment
+
+  // TODO: verify this is correct


Have we addressed this TODO?

echeipesh · 2017-03-08T20:27:23Z

-  val createSegment: Int => BitGeoTiffSegment = { i =>
-    val (segmentCols, segmentRows) = segmentLayout.getSegmentDimensions(i)
-    // val size = segmentCols * segmentRows
+  val decompressGeoTiffSegment = { (i: Int, bytes: Array[Byte]) =>


@lossyrob
This breaks "public" API by removing createSegment aside from having a bad name the signature of createSegment forces it to use getSegment, which breaks streaming.

Question: getSegment be added back for this PR with deprecation and a warning or is this a forgivable sin?

echeipesh · 2017-03-09T02:00:11Z

+  * The base trait of SegmentBytes. It can be implemented either as
+  * an Array[Array[Byte]] or as a ByteBuffer that is lazily read in.
+  */
+trait SegmentBytes extends Seq[Array[Byte]] {


This had to changes from Traversable to Seq because the only way the former can tell its length is to iterate through the collection, which is obviously not great for streaming.

echeipesh · 2017-03-09T02:04:46Z

Here is a handy test I'm using:

import geotrellis.spark.io.s3._
import geotrellis.spark.io._
import geotrellis.spark._
import geotrellis.raster._
import geotrellis.raster.io._
import geotrellis.raster.io.geotiff._
import geotrellis.proj4._
import geotrellis.vector._

val client = S3Client.DEFAULT

import geotrellis.spark.io.s3.util._
val rr = S3RangeReader("geotrellis-test", "rf-test/356f564e3a0dc9d15553c17cf4583f21-6.tif", client)
val tiff = MultibandGeoTiff.streaming(rr)
val sube = tiff.extent.center.buffer(tiff.extent.width * 0.01).envelope
tiff.crop(sube)

jbouffard and others added 30 commits December 5, 2016 12:14

Added StreamingSegmentBytes

e1b08c1

Signed-off-by: jbouffard <[email protected]>

Integrated StreamingSegmentBytes in SinglebandGeoTiffs

ccc9a67

Signed-off-by: jbouffard <[email protected]>

Integrated StreamingSegmentBytes in MultibandGeoTiffs

fb625c6

Signed-off-by: jbouffard <[email protected]>

Integrated StreamingSegmentBytes in GeoTiffReader

4bf33df

Signed-off-by: jbouffard <[email protected]>

Integrated StreamingSegmentBytes into Int16GeoTiffTile

0242cf6

Signed-off-by: jbouffard <[email protected]>

Integrated StreamingSegmentBytes into UByteGeoTiffTile

cc716f7

Signed-off-by: jbouffard <[email protected]>

Integrated StreamingSegmentBytes into Int16GeoTiffTile

ac2bac5

Signed-off-by: jbouffard <[email protected]>

Added StreamingSegmentBytesSpec

66c9a4d

Signed-off-by: jbouffard <[email protected]>

Integrated StreamingSegmentBytes into ByteGeoTiffTile

60fcd37

Signed-off-by: jbouffard <[email protected]>

Integrated StreamingSegmentBytes into UInt16GeoTiffTile

95b443b

Signed-off-by: jbouffard <[email protected]>

Integrated StreamingSegmentBytes into UInt32GeoTiffTile

5c0dde4

Signed-off-by: jbouffard <[email protected]>

Integrated StreamingSegmentBytes into Float32GeoTiffTile

03261f0

Signed-off-by: jbouffard <[email protected]>

Integrated StreamingSegmentBytes into Float64GeoTiffTile

6b4799f

Signed-off-by: jbouffard <[email protected]>

Added intersectingSegments to ArraySegmentBytes

70d3c35

Signed-off-by: jbouffard <[email protected]>

Removed LazySegmentBytes

f06d71f

Signed-off-by: jbouffard <[email protected]>

Added intersectingSegments to SegmentBytes

0b948bb

Signed-off-by: jbouffard <[email protected]>

Continuing debugging work

3c21091

Signed-off-by: jbouffard <[email protected]>

Continued to debug StreamingSegmentBytes

ebc1c42

Signed-off-by: jbouffard <[email protected]>

Removed LazySegmentBytes from SegmentBytesSpec

e930661

Signed-off-by: jbouffard <[email protected]>

Removed some debugging code and extranious methods from StreamingSegm…

ae257e8

…entBytes Signed-off-by: jbouffard <[email protected]>

Updated StreamingSegmentBytesSpec to reflect the changes to Streaming…

4c88e9e

…SegmentBytes Signed-off-by: jbouffard <[email protected]>

Moved MergeQueue from the Spark to Util

14a9562

Signed-off-by: jbouffard <[email protected]>

Updated the import path for MergeQueue in all files that use it

94428db

Signed-off-by: jbouffard <[email protected]>

Began work on reading consecutive segments as one chunk

46b46cc

Signed-off-by: jbouffard <[email protected]>

Removed bug that created duplicate chunks

dbd8f42

Signed-off-by: jbouffard <[email protected]>

Switched paramters around in ByteReaderExtensions

3f00aee

Signed-off-by: jbouffard <[email protected]>

Updated TiffTagsReader to reflect the changes in ByteReaderExtensions

3d0058c

Signed-off-by: jbouffard <[email protected]>

Continued to bug fix

011f041

Signed-off-by: jbouffard <[email protected]>

Fix streaming segment chunking

aae773a

Updated the various GeoTiffTiles crop methods to reflect the changes …

82c97e3

…made in StreamingSegmentBytes Signed-off-by: jbouffard <[email protected]>

Merge branch 'master' of github.com:pomadchin/geotrellis into chunky-…

2e53e9c

…streaming Signed-off-by: Grigory Pomadchin <[email protected]>

pomadchin force-pushed the chunky-streaming branch from 4b0a995 to 2e53e9c Compare March 2, 2017 07:59

pomadchin added 2 commits March 2, 2017 11:54

fix s3 unit tests

947131b

Signed-off-by: Grigory Pomadchin <[email protected]>

deinterleave functions fix

0d48955

Signed-off-by: Grigory Pomadchin <[email protected]>

pomadchin force-pushed the chunky-streaming branch from e19c4fd to 0d48955 Compare March 2, 2017 15:35

pomadchin added 6 commits March 2, 2017 20:17

improve bands and subset functions

71deb84

Signed-off-by: Grigory Pomadchin <[email protected]>

_combine(initValueHolder: ...) function refactor

ec4ccdc

Signed-off-by: Grigory Pomadchin <[email protected]>

combiners arity 2 refactor

d78fd98

Signed-off-by: Grigory Pomadchin <[email protected]>

use getSegments in GeoTiffWriter

1024743

Signed-off-by: Grigory Pomadchin <[email protected]>

multiband combiners improvements, moved boilerplate to use getSegment…

6e75560

…s function Signed-off-by: Grigory Pomadchin <[email protected]>

fix multiband combiners

ff0ec4b

Signed-off-by: Grigory Pomadchin <[email protected]>

pomadchin force-pushed the chunky-streaming branch from 8acb1f6 to ff0ec4b Compare March 2, 2017 19:46

restore LazySegmentBytes tests

e1258ec

Signed-off-by: Grigory Pomadchin <[email protected]>

pomadchin force-pushed the chunky-streaming branch from ab15c26 to e1258ec Compare March 3, 2017 14:58

lossyrob suggested changes Mar 8, 2017

View reviewed changes

echeipesh commented Mar 8, 2017

View reviewed changes

echeipesh force-pushed the chunky-streaming branch from 0bdcbb8 to 95c6a27 Compare March 9, 2017 01:52

echeipesh added 2 commits March 8, 2017 20:53

minimize API impact of the changes

95c6a27

Remove GeoTiffSegmenTransform.crop method until it is fully used

30f8597

echeipesh commented Mar 9, 2017

View reviewed changes

echeipesh removed the in progress label Mar 9, 2017

echeipesh mentioned this pull request Mar 9, 2017

Windowed GeoTiff Ingest raster-foundry/raster-foundry#1224

Merged

4 tasks done

lossyrob approved these changes Mar 9, 2017

View reviewed changes

echeipesh merged commit 796f651 into locationtech:master Mar 10, 2017

lossyrob mentioned this pull request Mar 12, 2017

Failure to stream GeoTiff window #1812

Closed

lossyrob modified the milestones: 1.1, 1.0.1 Mar 12, 2017

echeipesh added the api change label Mar 13, 2017

Navigation Menu

Repository navigation

Streaming/Windowed GeoTiff Reading - #1905

Streaming/Windowed GeoTiff Reading#1905
echeipesh merged 90 commits into
locationtech:masterfrom
echeipesh:chunky-streaming

echeipesh commented Dec 12, 2016 •

edited

Loading

Uh oh!

pomadchin commented Mar 2, 2017

Uh oh!

lossyrob Mar 7, 2017

Uh oh!

lossyrob Mar 7, 2017

Uh oh!

echeipesh Mar 8, 2017

Uh oh!

echeipesh Mar 9, 2017

Uh oh!

echeipesh commented Mar 9, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

echeipesh commented Dec 12, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pomadchin commented Mar 2, 2017

Uh oh!

lossyrob Mar 7, 2017

Choose a reason for hiding this comment

Uh oh!

lossyrob Mar 7, 2017

Choose a reason for hiding this comment

Uh oh!

echeipesh Mar 8, 2017

Choose a reason for hiding this comment

Uh oh!

echeipesh Mar 9, 2017

Choose a reason for hiding this comment

Uh oh!

echeipesh commented Mar 9, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

echeipesh commented Dec 12, 2016 •

edited

Loading