Skip to content

Added save action to ETL - #1764

Merged
echeipesh merged 5 commits into
locationtech:masterfrom
lossyrob:feature/etl-output-hook
Nov 7, 2016
Merged

echeipesh merged 5 commits into
locationtech:masterfrom
lossyrob:feature/etl-output-hook

Conversation

@lossyrob

@lossyrob lossyrob commented Nov 1, 2016 •

Copy link
Copy Markdown
Member

This feature allows us to include client code in the save process for ETL. Lets us do things like:

    EtlConf(args) foreach { conf =>
      val etl = Etl(conf)

      val sourceTiles = etl.load[ProjectedExtent, Tile]
      val (zoom, tiled) = etl.tile[ProjectedExtent, Tile, SpatialKey](sourceTiles)

      etl.save[SpatialKey, Tile](LayerId(etl.input.name, zoom), tiled, { (attributeStore, layerWriter, layerId, rdd) =>
        // Save off histogram computed from zoom 9, store in zoom 0's attributes.
        if(layerId.zoom == 9) {
          val histogram = rdd.histogram(512)
          attributeStore.write(
            layerId.copy(zoom = 0),
            "histogram",
            histogram: Histogram[Double])
        }
      })
    }

which is otherwise very difficult to do in the ETL context.

Also in this PR

Changing the layerExists functionality on AttributeStore to check specifically for the metadata attribute, so that you can store attributes in layer IDs without that layer being considered as existing.

@lossyrob
lossyrob force-pushed the feature/etl-output-hook branch from acac3ed to b751e2f Compare November 1, 2016 18:27
@lossyrob lossyrob added this to the 1.0 milestone Nov 1, 2016
@lossyrob lossyrob changed the title Added post save hook to ETL Added save action to ETL Nov 1, 2016
* @param rdd Tiled raster RDD with TileLayerMetadata
* @param id Layout ID to b
* @param rdd Tiled raster RDD with TileLayerMetadata
* @param postSave Function to allow saving additional attributes or layers per layer saved.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wrong comment name

@lossyrob

lossyrob commented Nov 3, 2016

Copy link
Copy Markdown
Member Author

@echeipesh your comment was addressed. GTG?

@lossyrob
lossyrob dismissed echeipesh’s stale review November 4, 2016 23:13

Made requested change

@echeipesh
echeipesh merged commit 6d49a7a into locationtech:master Nov 7, 2016
@lossyrob
lossyrob deleted the feature/etl-output-hook branch November 7, 2016 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants