Skip to content

Pyramid.fromLayerRDD fails to respect NODATA values with ResampleMethods.Max #3144

Closed
#3155
Closed
Pyramid.fromLayerRDD fails to respect NODATA values with ResampleMethods.Max#3144
#3155
Labels
bugrasterspark

Description

@CloudNiner
CloudNiner
opened on Oct 30, 2019

I have a TileLayerRDD[SpatialKey] and am generating an image pyramid with the following code snippet:

def savePng(
    layer: TileLayerRDD[SpatialKey],
    colorMap: ColorMap,
    outputPath: String
  ): Unit = {
    implicit val sc: SparkContext = layer.sparkContext

    val (baseZoom, baseLayer) = layer.reproject(ZoomedLayoutScheme(WebMercator, 256))
    val pyramid = Pyramid.fromLayerRDD(
      baseLayer,
      Some(baseZoom),
      Some(0),
      resampleMethod = ResampleMethods.Max
    )

    pyramid.levels.foreach { case (zoom, tileRdd) =>
      val imageRdd: RDD[(SpatialKey, Array[Byte])] =
        tileRdd.mapValues(_.renderPng(colorMap).bytes)

      val keyToPath = { k: SpatialKey => s"${outputPath}/${zoom}/${k.col}/${k.row}.png" }
      if (outputPath.startsWith("s3")) {
        SaveToS3(imageRdd, keyToPath, { request =>
          request.toBuilder.acl(ObjectCannedACL.PUBLIC_READ).build()
        })
      } else {
        SaveToHadoop(imageRdd, keyToPath)
      }
    }
  }

When this layer is generated and displayed on a map, all zoom levels past the base zoom level replace nodata values with the value of the first color ramp:
https://tilejson.io/g/90e81c9fd2ed500609829d5df5af93dc/view

When I use the default ResampleMethod.Average this does not occur. It appears that MaxResample handles NoData values correctly

Activity

  1. CloudNiner
    added
    bug
    raster
    spark
    on Oct 30, 2019

  2. CloudNiner commented on Oct 30, 2019

    @CloudNiner
    CloudNiner
    on Oct 30, 2019
    ContributorAuthor

    Here's two images at approximately the same location that demonstrate the issue (if the tilejson link ever disappears):

    No data filled in as we zoom out

    Screen Shot 2019-10-30 at 4 47 25 PM

    Correct for basezoom

    Screen Shot 2019-10-30 at 4 47 22 PM

  3. echeipesh commented on Nov 15, 2019

    @echeipesh
    echeipesh
    on Nov 15, 2019
    Contributor
    scala> val tile = ArrayTile.empty(DoubleConstantNoDataCellType, 4, 4)
tile: geotrellis.raster.MutableArrayTile = DoubleConstantNoDataArrayTile([D@708afe26,4,4)

scala> tile.asciiDrawDouble()
res13: String =
"    ND    ND    ND    ND
    ND    ND    ND    ND
    ND    ND    ND    ND
    ND    ND    ND    ND

"

scala> tile.resample(2, 2, ResampleMethods.Max).asciiDrawDouble()
res14: String =
" -2.147483648E9 -2.147483648E9
 -2.147483648E9 -2.147483648E9

"

    Note, its important to use asciiDrawDouble because asciiDraw obscures the problem.

    As @kervel pointed out on gitter:

        val doubleMax = indices.foldLeft(Double.MinValue) { case (currentMax, coords) =>
      val v = tile.getDouble(coords._1, coords._2)
      // Double.NaN would *always* be max
      if (isData(v)) math.max(currentMax, v) else currentMax
    }
    if (doubleMax == Double.MinValue) NODATA else doubleMax

    here NODATA is the Int nodata... i think it should be doubleNODATA

  4. echeipesh
    closed this as completedin #3155on Nov 16, 2019
  5. echeipesh
    added a commit that references this issue on Nov 16, 2019

    MinResample / MaxResample: preserve NODATA correctly for double cell …

    4b918e0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugrasterspark

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions