Repository navigation
Fix spatial join when left and right rdd have different PartitionerIndex - #3175
Merged
Merged
Conversation
soxofaan
force-pushed
the
issue3168-spatial-join
branch
5 times, most recently
from
January 10, 2020 11:30
9920fac to
df7be80
Compare
…ted) Signed-off-by: Stefaan Lippens <[email protected]>
…dex in ReorderedRDD and SpacePartitioner Signed-off-by: Stefaan Lippens <[email protected]>
Signed-off-by: Stefaan Lippens <[email protected]>
soxofaan
force-pushed
the
issue3168-spatial-join
branch
from
January 10, 2020 11:35
df7be80 to
d77637f
Compare
echeipesh
added a commit
that referenced
this pull request
Apr 6, 2020
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Fixes issue described at #3168
The original implementation blindly assumed same partition indexing, causing empty join when using custom partitioning with one of the RDDs being joined.
This PR adds tests for this use case and fixes the issue by adding an additional check that same PartitionIndexer is used, so that reshuffle is triggered when that is not the case.
PR also fixed some inactive asserts in same file.
It also addresses partitioning mismatch in ReorderedSpaceRDD
Checklist
Module Hierarcy updated, if necessarydocsguides update, if necessaryNew user API has useful Scaladoc stringsCloses #3168