Repository navigation
[SPARK-30082][SQL] Depend on Scala type coercion when building replace query - #628
Merged
bulldozer-bot[bot] merged 1 commit intoJan 15, 2020
Merged
Conversation
Member
|
Thanks for your interest in palantir/spark, @johnhany97! Before we can accept your pull request, you need to sign our contributor license agreement - just visit https://cla.palantir.com/ and follow the instructions. Once you sign, I'll automatically update this pull request. |
johnhany97
force-pushed
the
SPARK-30082-palantir
branch
from
December 2, 2019 16:10
6d31ede to
a491e1f
Compare
Member
Author
|
@mccheah, can you take a look? |
### What changes were proposed in this pull request?
Do not cast `NaN` to an `Integer`, `Long`, `Short` or `Byte`. This is because casting `NaN` to those types results in a `0` which erroneously replaces `0`s while only `NaN`s should be replaced.
### Why are the changes needed?
This Scala code snippet:
```
import scala.math;
println(Double.NaN.toLong)
```
returns `0` which is problematic as if you run the following Spark code, `0`s get replaced as well:
```
>>> df = spark.createDataFrame([(1.0, 0), (0.0, 3), (float('nan'), 0)], ("index", "value"))
>>> df.show()
+-----+-----+
|index|value|
+-----+-----+
| 1.0| 0|
| 0.0| 3|
| NaN| 0|
+-----+-----+
>>> df.replace(float('nan'), 2).show()
+-----+-----+
|index|value|
+-----+-----+
| 1.0| 2|
| 0.0| 3|
| 2.0| 2|
+-----+-----+
```
### Does this PR introduce any user-facing change?
Yes, after the PR, running the same above code snippet returns the correct expected results:
```
>>> df = spark.createDataFrame([(1.0, 0), (0.0, 3), (float('nan'), 0)], ("index", "value"))
>>> df.show()
+-----+-----+
|index|value|
+-----+-----+
| 1.0| 0|
| 0.0| 3|
| NaN| 0|
+-----+-----+
>>> df.replace(float('nan'), 2).show()
+-----+-----+
|index|value|
+-----+-----+
| 1.0| 0|
| 0.0| 3|
| 2.0| 0|
+-----+-----+
```
### How was this patch tested?
Added unit tests to verify replacing `NaN` only affects columns of type `Float` and `Double`
Closes apache#26749 from johnhany97/SPARK-30082-2.4.
Authored-by: John Ayad <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
johnhany97
force-pushed
the
SPARK-30082-palantir
branch
from
January 15, 2020 14:00
ea4d09e to
82e0669
Compare
lwwmanning
approved these changes
Jan 15, 2020
|
Approving as this is a cherry-pick |
mccheah
approved these changes
Jan 15, 2020
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
apache#26738
apache#26749
What changes were proposed in this pull request?
Depend on type coercion when building the replace query. This would solve an edge case where when trying to replace
NaNs,0s would get replace too.Why are the changes needed?
This Scala code snippet:
returns
0which is problematic as if you run the following Spark code,0s get replaced as well:Does this PR introduce any user-facing change?
Yes, after the PR, running the same above code snippet returns the correct expected results:
And additionally, query results are changed as a result of the change in depending on scala's type coercion rules.
How was this patch tested?
Added unit tests to verify replacing
NaNonly affects columns of typeFloatandDouble.