Repository navigation
VectorTiles Part 2.1: The Reckoning (again) - #1622
Merged
lossyrob merged 95 commits intoOct 5, 2016
Merged
Conversation
…types into the feature category
… future, added better filter funcitonality, added more test data and test data generating examples
…elevation into vector-tiles-decoder
Vector tiles decoder
…is into feature/vector-tiles
- This is to make for easier addition of backends
- These traits don't assume a backend, and should first be extended by classes like `ProtobufTile`, etc.
- Using lazy Streams here allows us to avoid strictly holding the Stream head,
meaning no Features of a geomtype we don't care about will be parsed,
unless we ask.
- This is advantangeous for queries as well. If you are looking for a Feature
match on some metadata point, only Features will be parsed until you find
what you're looking for.
- Potential disadvantage being intermittent instances of the opposite
Single/Multi you're looking for will fully parse as well. The alternative
is the ad-hoc reimplementation of laziness with internal mutable data
structures.
Consider the following scenarios, where P and MP are Polygon and
MultiPolygon respectively:
[P MP P MP P MP] -- A list of alternating raw Polygon features.
(1) The user wants to find a particular Polygon, which unknown to them
is the second one in the list. They have to parse the first P,
do *something* to the first MP, parse the second P and match on it,
then stop.
With Streams, the original list now looks like: [ MP P MP ]
With custom laziness, it looks like: [ MP MP P MP ]
The custom laziness wins for speed here, since we were able to
cancel the parsing of the first MP early, and the Streams
approach fully parsed the first MP.
(2) The user wants to perform another operation, this time across all
Ps. Both approaches must thus map over the entire list.
With Streams, the original list is now empty: []
With custom laziness, it looks like: [ MP MP MP ]
It's harder to tell who wins here, since while the Streams
had to waste time fully parsing each MP, the custom laziness
had to reparse old MPs it had already looked through.
(3) The user wants to perform another operation on all Ps. The streams
can go ahead since everything has been parsed. The custom approach
must reparse all the MPs to check for Ps, since it wouldn't know
there weren't any left.
(4) The user wants to perform an operation on MPs this time. The streams
can go ahead since the MPs are already parsed. The custom approach
has to reparse the MPs for the fourth time.
My takeaway: the custom approach is better for one-off operations on a
particular geometry type. The stream approach quickly overtakes the other
if you plan multiple operations over the same geometries.
- This reduces redundant code, and takes into account how the `Seq[Command]`
will actually be traversed (that is, fairly agnostically as to what true
Geometry type lies beneath). This way, one doesn't have to "backtrack" when
parsing Multi{Line,Polygon}s, or try any analytics to discover which true
Geometry you have before you attempt actual parsing.
- And so the only thing require we during IO is that Extent. This keeps the API simple.
- Renamed to avoid confusion with GeoTrellis `Extent`.
- This shaves off a bit more time from the vanilla decoding process
- Something is wrong with the MultiPolygons...
- Avoiding borrowing the Tuple schema, it was causing problems.
Contributor
Author
|
Rebased off of the lastest additions to the original PR. |
- It handles the embedded `Extent` properly now.
- This avoids a Spark exception.
- Since VectorTiles encode their bytes themselves to reduce complexity.
Contributor
Author
|
A demo of this functionality can be found here: https://github.com/fosskers/vectortile-io |
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TODO
Motivation
Decoding and encoding individual tiles is fine, but more work is necessary to glue VectorTiles into the rest of GeoTrellis.
This builds off #1563 .