Skip to content

ColorMap => String - #1512

Merged
lossyrob merged 6 commits into
locationtech:masterfrom
fosskers:enhancement/colormap-to-string
Jun 14, 2016
Merged

lossyrob merged 6 commits into
locationtech:masterfrom
fosskers:enhancement/colormap-to-string

Conversation

@fosskers

@fosskers fosskers commented Jun 7, 2016 •

Copy link
Copy Markdown
Contributor

TODO

  • Basic functionality
  • Move breaksString into trait and inherit
  • Special definition for IntCachedColorMap
  • Tests

Motivation

When using the render output module in the ETL process, one must provide a breaks argument, or all tiles will be rendered grayscale. Example:

--output render -O encoding=png path=file:///home/colin/tiles/{name}/{z}-{x}-{y}.png breaks="23:cc00ccff;30:aa00aaff;120:ff0000ff"

The problem is that for datasets which don't have well-defined colour breaks ahead of time like nlcd, there's no way to know what your break string should be for input into ETL. A ColorMap object can be created easily given RDD.colorBreaks and then rendered to a PNG within Scala code, but there is no way to know what String would represent that ColorMap from the outside. This chicken-and-egg problem requires a greater fix to how ETL accepts breaks information from the outside, but this is a start.

- Fairly hacky, but demonstrates it's possible.
@fosskers

fosskers commented Jun 7, 2016

Copy link
Copy Markdown
Contributor Author

To test:

sbt
project raster
console
import geotrellis.raster.render.ColorMap
val m = ColorMap.fromString("23:cc00ccff;30:aa00aaff;120:ff0000ff").get
ColorMap.breaksString(m.breaksMap)

private lazy val orderedColors: Vector[Int] = orderedBreaks.map(breaksToColors(_))
lazy val colors = orderedColors
lazy val breaksMap = breaksToColors
lazy val breaksMapDouble = Map.empty[Double,Int]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need a breaks map? Can't we just implement this functionality as the toString method?

@fosskers fosskers Jun 7, 2016 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Temporary hack, see here

@fosskers

fosskers commented Jun 7, 2016 •

Copy link
Copy Markdown
Contributor Author

Really, the best thing would be for ColorMap to be polymorphic on its internal number type:

trait ColorMap[T] extends Serializable { ... }

And then duplicate code between IntColorMap and DoubleColorMap could be removed. Then, the ColorMap trait could ask for a def breaksMap: Map[T,Int], and breaksString could be moved into the trait defined as:

def breaksString: String = {
  breaksMap  // was `breaksToColors` passed in as an argument
       .toStream
       .map({ case (k,v) => s"${k}:${Integer.toHexString(v)}"})                                           
       .mkString(";")
}

The huge downside of this would be that client code would have to care about the parameterized ColorMap[T], unless we used some cool type aliasing so nothing breaks.
Thoughts, @lossyrob @echeipesh ?

@lossyrob why I'm not in favour of just defining the .toString this way is because it's considered bad form in Haskell-land to have custom output for the show :: Show t => t -> String function, which is similar to toString. Typically show gives you the raw serialized form of the object, such that:

read . show == id

and other "pretty printing" functions are given obvious names.

@fosskers

fosskers commented Jun 7, 2016

Copy link
Copy Markdown
Contributor Author

Another trip-up: the user-level API doesn't expose IntColorMap, etc, only ColorMap. The colouring function has to be visible to the trait. You could put it in the companion object (as I've done initially here) but then the arguments to it must also be visible to the trait. IntCachedColorMap throws a wrench into that, as it has no Map internally, but still extends ColorMap.

@lossyrob

lossyrob commented Jun 7, 2016

Copy link
Copy Markdown
Member

Your hitting up against the need for us not to abstract over Int and Double typed things due to performance implications. We jump through hoops to have Double and Int versions of things because if we try to make it generic, things slow down in very painful ways - boxing is the enemy, and ColorMaps are performance critical code that must not box.

I'm not sure "it not being good form in Haskell" is a good reason to avoid using the method that's there, defined as returning a string representation of the object. One could argue that the string representation of a color map that you are concocting is actually a good representation for a user trying to read what the color map is; however, even if we wanted to differentiate between a "show" toString and some other form (def breaksString: String), we could still use implementations in the concrete classes to take care of the Double vs Int bits and avoid the generic breaksString method you have.

@fosskers

fosskers commented Jun 7, 2016 •

Copy link
Copy Markdown
Contributor Author

haskell

That was more of an explanation as to my natural want to avoid that pattern, not necessarily that we should do it that way "cause Haskell". The question becomes, where do we want the isomorphism? Between ColorMap.fromString <-> breaksString or ColorMap.fromString <-> toString?

re: my last comment. override toString could work for the inheritance problems. I'm not a fan of the duplicate code, but if this is performance critical than I can see the need.

@lossyrob

lossyrob commented Jun 7, 2016

Copy link
Copy Markdown
Member

Gotcha. Yeah I think the argument could be made though that we would want to avoid using toString because it's too generic of a thing, and instead use the breakString with inheritance.

@fosskers

fosskers commented Jun 7, 2016

Copy link
Copy Markdown
Contributor Author

Meaning the actual definition of breaksString will be duplicated between the *ColorMap classes for performance reasons? I'll do it that way if that's the prevailing pattern.

@lossyrob

lossyrob commented Jun 7, 2016

Copy link
Copy Markdown
Member

More to do with why we can't do

trait ColorMap[T] extends Serializable { ... }

and be generic on T.

But code duplication in this case might be tough to avoid, since the code to deal with doubles and ints will look the same (but not with the cached version of int, which you mentioned above).

@fosskers fosskers changed the title [WIP] ColorMap => String ColorMap => String Jun 7, 2016
new IntCachedColorMap(orderedColors, ch, options)
}

lazy val breaksString: String = {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why lazy val for this and the other one?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the same reason that orderedColors is lazy, so that these potentially unneeded values aren't calculated upon ColorMap instantiation.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't thinking vals...why not defs? Is there ever a reason to hold them in memory after calculation (e.g. its an expensive operation that will be reused many times to justify the instance footprint)?

@fosskers fosskers Jun 13, 2016 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't see it being called over and over for one instance of a ColorMap. I suppose the GC behaviour for these are:

  • val: Eagerly evaluate, keep in memory
  • lazy val: Lazily evaluate, and then keep in memory
  • def: Lazily evaluate, don't keep in memory

If so, then perhaps it's best as a def.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed.
+1 after this change.

@fosskers

Copy link
Copy Markdown
Contributor Author

Good to go, @lossyrob .

@lossyrob
lossyrob merged commit 42ab052 into locationtech:master Jun 14, 2016
@lossyrob lossyrob added this to the 1.0 milestone Oct 18, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants