RangeReader SPI by jbouffard · Pull Request #2977 · locationtech/geotrellis

jbouffard · 2019-06-11T19:59:25Z

Overview

This PR provides an SPI API for RangeReader for the following backeds: file, hdfs, http, and s3.

Checklist

docs/CHANGELOG.rst updated, if necessary
docs guides update, if necessary
New user API has useful Scaladoc strings
Unit tests added for bug-fix or new feature

Demo

val rangeReader: FileRangeReader = RangeReader(new URI("file///tmp/catalog/file.tif"))

Notes

The HdfsRangeReader has a problem where it will use a default Configuration even if another one is present. This is because there is no current way of getting a Configuration without it being provided explicitly or found via the SparkContext. HdfsConfig introduced in geotrellis/geotrellis-contrib#186 would resolve this issue.

Closes #2940

…the (locationtech#2927) spark package into it Signed-off-by: Jacob Bouffard <[email protected]> Added the apacheIO dependency to the layers project Signed-off-by: Jacob Bouffard <[email protected]> Made the spark package depend on the layers package Signed-off-by: Jacob Bouffard <[email protected]> Moved the avro package to geotrellis.layers.io and updated the imports Signed-off-by: Jacob Bouffard <[email protected]> Moved the index package to geotrellis.layers.io and updated the imports Signed-off-by: Jacob Bouffard <[email protected]> Moved the json package to geotrellis.layers.io and updated some of the imports Signed-off-by: Jacob Bouffard <[email protected]> Moved Metadata, TileLayerMetadata, LayerId, and Mergable from spark and spark.merge to layers and layers.merge, respectively. In addition, imports were updated Signed-off-by: Jacob Bouffard <[email protected]> Moved the io, cog, file, and hadoop logic from geotrellis.spark.io into geotrellis.layers.io Signed-off-by: Jacob Bouffard <[email protected]> Moved buffer and mapalgebra logic from spark.buffer and spark.mapalgebra to layers.buffer and layers.mapalbegra, respectively Signed-off-by: Jacob Bouffard <[email protected]> Moved the Collection mask methods from spark.mask to layer.mask Signed-off-by: Jacob Bouffard <[email protected]> Broke out collections API spark into the layers package Signed-off-by: Jacob Bouffard <[email protected]> Added application.conf to layers tests' resources Signed-off-by: Jacob Bouffard <[email protected]> Refactored the BufferTiles object Signed-off-by: Jacob Bouffard <[email protected]> Refactored the Mask object Signed-off-by: Jacob Bouffard <[email protected]> Cleaned and added tests Signed-off-by: Jacob Bouffard <[email protected]> Moved BufferedTile and BufferSizes to the raster package Signed-off-by: Jacob Bouffard <[email protected]> Removed the mapalgebra tests from layers Signed-off-by: Jacob Bouffard <[email protected]> Removed the io package in layers Signed-off-by: Jacob Bouffard <[email protected]> Cleaned up the hadoop formats logic Signed-off-by: Jacob Bouffard <[email protected]> Moved the KeyJsonFormats to the tiling project Signed-off-by: Jacob Bouffard <[email protected]> Updated the spark-testkit Signed-off-by: Jacob Bouffard <[email protected]> Updated and fixes the spark tests so that they can compile Signed-off-by: Jacob Bouffard <[email protected]> Continued to try and get the spark tests working Signed-off-by: Jacob Bouffard <[email protected]> Continued to woking on the spark tests Signed-off-by: Jacob Bouffard <[email protected]> Updated the tests so that they all passed Signed-off-by: Jacob Bouffard <[email protected]> Cleaned up the hadoop backend packages Signed-off-by: Jacob Bouffard <[email protected]> Moved CamelCaseConfig to geotrellis.util Signed-off-by: Jacob Bouffard <[email protected]> Renamed the njoin methods parJoin and moved them into IOUtils Signed-off-by: Jacob Bouffard <[email protected]> Moved HdfsUtilsSpec back to spark tests Signed-off-by: Jacob Bouffard <[email protected]> Renamed TileLayerRDDMetadata to CollectTileLayerMetadata Signed-off-by: Jacob Bouffard <[email protected]> Renamed the RDDLayerProviders to SparkLayerProviders Signed-off-by: Jacob Bouffard <[email protected]> Renamed the spark resources Signed-off-by: Jacob Bouffard <[email protected]> Added a TODO Signed-off-by: Jacob Bouffard <[email protected]> Split the accumulo package into two new ones: accumulo and layers-accumulo. The former contains logic that uses Spark while the latter does not Signed-off-by: Jacob Bouffard <[email protected]> Added a reference.conf to the layers-accumulo package Signed-off-by: Jacob Bouffard <[email protected]> Updated the accumulo tests Signed-off-by: Jacob Bouffard <[email protected]> Updated the CHANGELOG Signed-off-by: Jacob Bouffard <[email protected]> Added layers-accumulo to the root project Signed-off-by: Jacob Bouffard <[email protected]> Added layers-accumulo to the various scripts Signed-off-by: Jacob Bouffard <[email protected]> Updated the geowave project Signed-off-by: Jacob Bouffard <[email protected]> Split the cassandra package into two new ones: cassandra and layers-cassandra. The former contains logic that uses Spark while the latter does not Signed-off-by: Jacob Bouffard <[email protected]> Updated the cassandra tests Signed-off-by: Jacob Bouffard <[email protected]> Removed unneeded wrapper class Signed-off-by: Jacob Bouffard <[email protected]> Added layers-cassandra to the root project Signed-off-by: Jacob Bouffard <[email protected]> Added layers-cassandra to the various scripts Signed-off-by: Jacob Bouffard <[email protected]> Split HBase spark/non spark code Organize hbase store project Delete old hbase dir Add hbase collection layer provider Update META-INF for SPI Rename layer provider for spark case Split the s3 package into two new ones: s3 and s3-store. The former contains logic that uses Spark while the latter does not Signed-off-by: Jacob Bouffard <[email protected]> Updated the SPIs for the s3 and s3-store packages Signed-off-by: Jacob Bouffard <[email protected]> Cleaned up the hbase packages Signed-off-by: Jacob Bouffard <[email protected]> Updated the spark-pipeline project Signed-off-by: Jacob Bouffard <[email protected]> Updated the doc-examples project Signed-off-by: Jacob Bouffard <[email protected]> Fixed imports Signed-off-by: Jacob Bouffard <[email protected]> Renamed the cassandra-layers and cassandra packages to cassandra-store and cassandra-spark, respectively Signed-off-by: Jacob Bouffard <[email protected]> Updated in the imports for the cassandra-spark console Signed-off-by: Jacob Bouffard <[email protected]> Fixed docker image name Signed-off-by: Jacob Bouffard <[email protected]> Bumped the downloaded Hbase version to 2.1.5 Signed-off-by: Jacob Bouffard <[email protected]> Renamed the accumulo-layers and accumulo packages to accumulo-store and accumulo-spark, respectively Signed-off-by: Jacob Bouffard <[email protected]> Renamed the class path in the accumulo-spark package to geotrellis.spark.store.accumulo Signed-off-by: Jacob Bouffard <[email protected]> Updated in the imports for the accumulo-spark console Signed-off-by: Jacob Bouffard <[email protected]> Cleaned up after rebase Signed-off-by: Jacob Bouffard <[email protected]> Fixed the serialization in HadoopAttributeStore Signed-off-by: Jacob Bouffard <[email protected]> Changed the constructors for the HadoopAttributeStore Signed-off-by: Jacob Bouffard <[email protected]> Updated the imports in the geowave package Signed-off-by: Jacob Bouffard <[email protected]> Fixed the resources in the hbase-spark project Signed-off-by: Jacob Bouffard <[email protected]> Fixed the layerIdString method in HBaseAttributeStore Signed-off-by: Jacob Bouffard <[email protected]> Renamed spark.io to spark.store Signed-off-by: Jacob Bouffard <[email protected]> Renamed the s3 package to s3-spark Signed-off-by: Jacob Bouffard <[email protected]> Updated the dependencies Signed-off-by: Jacob Bouffard <[email protected]> Fixed the container name for cassandra Signed-off-by: Jacob Bouffard <[email protected]> Fixed failing test Signed-off-by: Jacob Bouffard <[email protected]>

Update ingest documentation Update docs; scripts; changelog Update heading in pipeline docs Reremove etl Remove lingering references to deprecated ETL Fixed bad rebase Signed-off-by: Jacob Bouffard <[email protected]>

Signed-off-by: Jacob Bouffard <[email protected]>

moradology · 2019-06-12T15:24:23Z

Take a look at the provider here - hdfs+ isn't valid outside of GT but we use it to provide clues to SPI

moradology

Looking pretty good. There are some spots that might be worth reflecting on that I've called out

Signed-off-by: Jacob Bouffard <[email protected]>

pomadchin

I fixed tests, check out d7bacc8, I left only a single comment; also there are no headers in all new files. You can generate headers using headerCreate command and test:headerCreate for tests.

Also tests folders structure doesn't follow the package namings (the real question is why all the tests are in /src/test/scala/geotrellis/spark/io/s3/?) and there is no CHANGELOG.

pomadchin · 2019-06-13T01:13:23Z

+
+import org.scalatest._
+
+class S3RangeReaderProviderSpec extends FunSpec with Matchers {


Why this file is in s3-spark and not in s3-store? The implementation of the S3RangeReaderProvider is in s3-store.

There wasn't a really great reason. We just have all of our tests for s3-store in s3-spark still, so I just put it there.

@jbouffard can you look into the tests structure more?

@pomadchin Actually, we can't move it to s3-store as it needs TestUtils which is in s3-spark.

pomadchin · 2019-06-15T22:19:52Z

@jbouffard can you rebase this PR?

jbouffard · 2019-06-17T14:16:00Z

Superseded by #2998

jbouffard and others added 5 commits June 7, 2019 16:23

Remove ETL

29f80fe

Update ingest documentation Update docs; scripts; changelog Update heading in pipeline docs Reremove etl Remove lingering references to deprecated ETL Fixed bad rebase Signed-off-by: Jacob Bouffard <[email protected]>

Created RangeReader SPI for the file, http, and s3 backends

dfc8df8

Signed-off-by: Jacob Bouffard <[email protected]>

Implemented RangeReader SPI for hdfs

04a1787

Signed-off-by: Jacob Bouffard <[email protected]>

Updated imports

e66aa5a

Signed-off-by: Jacob Bouffard <[email protected]>

moradology suggested changes Jun 12, 2019

View reviewed changes

Comment thread layers/src/main/scala/geotrellis/layers/hadoop/HadoopCollectionLayerProvider.scala Outdated

Comment thread layers/src/main/scala/geotrellis/layers/hadoop/util/HdfsRangeReaderProvider.scala Outdated

Comment thread project/Dependencies.scala Outdated

Cleaned up code based on review feedback

45d47d3

Signed-off-by: Jacob Bouffard <[email protected]>

pomadchin added the in progress label Jun 12, 2019

pomadchin force-pushed the feature/range-reader-spi branch 2 times, most recently from d59a745 to 5828659 Compare June 13, 2019 01:07

pomadchin requested changes Jun 13, 2019

View reviewed changes

pomadchin force-pushed the feature/range-reader-spi branch from 5828659 to 3ec17b7 Compare June 13, 2019 01:43

pomadchin reviewed Jun 13, 2019

View reviewed changes

Comment thread s3-spark/src/test/scala/geotrellis/spark/io/s3/util/S3RangeReaderProviderSpec.scala

Fix S3Provider specs

d7bacc8

pomadchin force-pushed the feature/range-reader-spi branch from 3ec17b7 to d7bacc8 Compare June 13, 2019 02:26

pomadchin reviewed Jun 13, 2019

View reviewed changes

Comment thread s3-store/src/main/scala/geotrellis/store/s3/util/S3RangeReader.scala

echeipesh force-pushed the feature/spark-reorganization branch 5 times, most recently from d5aa869 to 5af6e80 Compare June 15, 2019 19:13

pomadchin assigned jbouffard Jun 15, 2019

jbouffard mentioned this pull request Jun 17, 2019

RangeReader SPI #2998

Merged

3 tasks done

jbouffard closed this Jun 17, 2019

pomadchin removed the in progress label Oct 3, 2019

Navigation Menu

Repository navigation

RangeReader SPI - #2977

RangeReader SPI#2977
jbouffard wants to merge 7 commits into
locationtech:feature/spark-reorganizationfrom
jbouffard:feature/range-reader-spi

jbouffard commented Jun 11, 2019

Uh oh!

moradology commented Jun 12, 2019

Uh oh!

moradology left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pomadchin left a comment •

edited

Loading

Uh oh!

pomadchin Jun 13, 2019

Uh oh!

jbouffard Jun 13, 2019

Uh oh!

pomadchin Jun 14, 2019

Uh oh!

jbouffard Jun 14, 2019

Uh oh!

Uh oh!

Uh oh!

pomadchin commented Jun 15, 2019

Uh oh!

jbouffard commented Jun 17, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		import org.scalatest._

		class S3RangeReaderProviderSpec extends FunSpec with Matchers {

Conversation

jbouffard commented Jun 11, 2019

Overview

Checklist

Demo

Notes

Uh oh!

moradology commented Jun 12, 2019

Uh oh!

moradology left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pomadchin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pomadchin Jun 13, 2019

Choose a reason for hiding this comment

Uh oh!

jbouffard Jun 13, 2019

Choose a reason for hiding this comment

Uh oh!

pomadchin Jun 14, 2019

Choose a reason for hiding this comment

Uh oh!

jbouffard Jun 14, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pomadchin commented Jun 15, 2019

Uh oh!

jbouffard commented Jun 17, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pomadchin left a comment •

edited

Loading