Skip to content

Move to Spark 2; Scala 2.10 deprecation - #1628

Merged
lossyrob merged 8 commits into
locationtech:masterfrom
pomadchin:feature/spark-2
Sep 23, 2016
Merged

Move to Spark 2; Scala 2.10 deprecation#1628
lossyrob merged 8 commits into
locationtech:masterfrom
pomadchin:feature/spark-2

Conversation

@pomadchin

@pomadchin pomadchin commented Sep 9, 2016 •
edited
Loading

Copy link
Copy Markdown
Member

Tests are passing, needs to be tested on a real cluster, fixes #1626

  • scala 2.10 deprecated
  • all libs update
  • Travis CI
  • Landsat EMR demo test
  • remove logging usage from serializable code (private Spark 2.0 Logging overload) dep hell related to scala landsat util (Scala 2.10 deprecation azavea/scala-landsat-util#6)
  • if we are going to support both Java 7 and Java 8, it is not possible to update typesafe config dep
  • noticed a bug / feature trying to update geotools
val actual = GeometryToSimpleFeature(multiPolygon, None, emptyList)
val expected = GeometryToSimpleFeature(multiPolygon, None, emptyList)

actual.equals(expected) 
// throws com.vividsolutions.jts.geom.TopologyException: 
// side location conflict [ (0.0, 3.0, NaN) ]

@pomadchin

pomadchin commented Sep 9, 2016 •
edited
Loading

Copy link
Copy Markdown
Member Author

It is still unclear for me, why we still need to exclude kryo (otherwise hbase tests would fail):

"de.javakaffee" % "kryo-serializers" % "0.38" exclude("com.esotericsoftware.kryo", "kryo") exclude("com.esotericsoftware", "kryo"),
"com.esotericsoftware" % "kryo-shaded" % "3.0.3"

@pomadchin pomadchin changed the title Move to Spark 2 Move to Spark 2 and Scala 2.10 deprecation Sep 12, 2016
@pomadchin pomadchin changed the title Move to Spark 2 and Scala 2.10 deprecation Move to Spark 2; Scala 2.10 deprecation Sep 12, 2016
@pomadchin

pomadchin commented Sep 19, 2016

Copy link
Copy Markdown
Member Author

Should we deprecate java 7 as well?

@pomadchin pomadchin mentioned this pull request Sep 20, 2016
Closed
@lossyrob

lossyrob commented Sep 23, 2016

Copy link
Copy Markdown
Member

@pomadchin re: java 7, I belive we decided yes, if there's changes that are required for that they can go in another PR.

@lossyrob
lossyrob merged commit 0a377df into locationtech:master Sep 23, 2016
This was referenced Sep 23, 2016
Merged
Closed
@lossyrob lossyrob added this to the 1.0 milestone Oct 18, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

1.0

Development

Successfully merging this pull request may close these issues.

SparkUtils not build for 2.0

2 participants

@pomadchin @lossyrob