AWS SDK v2 by moradology · Pull Request #2911 · locationtech/geotrellis

moradology · 2019-04-25T21:11:22Z

Overview

This PR removes v1 of the AWS SDK in favor of v2 and removes dependence upon the (Geotrellis provided) S3Client wrapper which was used to ease tests via MockS3Client. Work in the tests remains to implement a MockS3Client that simply extends the (Amazon provided as of v2 of the SDK) S3Client interface.

Tests aren't yet ready for the show as work remains to actually implement a MockS3Client before this can be merged but things are compiling and should work as before

Checklist

docs/CHANGELOG.rst updated, if necessary
Unit tests added for bug-fix or new feature
Test on EMR cluster

Notes

Some semantic differences between v1 and v2 need to be explored. Most pressingly, with v1 it was fine practice to spawn clients as necessary. This is no longer true. As of v2, it is recommended that clients be reused throughout the life of an application. This PR hasn't attempted to characterize or fix any problems arising from spawning clients. See this for more

Closes #2466
Closes #2302

pomadchin · 2019-05-15T03:45:50Z

-        "com.fasterxml.jackson.core" % "jackson-core" % "2.6.5",
-        "com.fasterxml.jackson.core" % "jackson-databind" % "2.6.5",
-        "com.fasterxml.jackson.core" % "jackson-annotations" % "2.6.5"
+        "com.fasterxml.jackson.core" % "jackson-core" % "2.6.7",


This update needs to be double checked on EMR cluster; see lucidworks/spark-solr#179

moradology · 2019-05-21T14:18:56Z

Probably worth looking through while reviewing/reading any of this code: https://github.com/aws/aws-sdk-java-v2/blob/master/docs/LaunchChangelog.md

pomadchin · 2019-05-24T17:01:51Z

software.amazon.awssdk 2.5.49 CQ 19892
com.fasterxml.jackson.core:jackson-databind:2.6.7 PB CQ 19887
com.fasterxml.jackson.core:jackson-annotations:2.6.7 PB CQ 19888
com.fasterxml.jackson.core:jackson-core:2.6.7 PB CQ 19889
org.reactivestreams:reactive-streams:1.0.2 PB CQ 19890
com.typesafe.netty:netty-reactive-streams-http:2.0.0 CQ 19893
io.netty:netty-all:4.1.33.Final CQ 19895

pomadchin · 2019-05-24T17:20:05Z

There are still some exceptions happen during the ingest:

19/05/24 17:14:29 ERROR Executor: Exception in task 802.0 in stage 9.0 (TID 20854)
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Read timed out
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.handleThrownException(RetryableStage.java:136)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:94)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:62)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:57)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:240)
	at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:96)
	at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:120)
	at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:73)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:44)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
	at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:3053)
	at geotrellis.spark.io.s3.S3RDDWriter$$anonfun$update$1$$anonfun$geotrellis$spark$io$s3$S3RDDWriter$$anonfun$$retire$1$1.apply(S3RDDWriter.scala:139)
	at geotrellis.spark.io.s3.S3RDDWriter$$anonfun$update$1$$anonfun$geotrellis$spark$io$s3$S3RDDWriter$$anonfun$$retire$1$1.apply(S3RDDWriter.scala:139)
	at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:87)
	at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:351)
	at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:372)
	at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:312)
	at cats.effect.internals.IOShift$Tick.run(IOShift.scala:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:233)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:102)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:214)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:66)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:51)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:35)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:64)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.doExecute(RetryableStage.java:113)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:86)
	... 29 more

pomadchin · 2019-05-24T17:50:55Z

It looks acceptable though; I would say it can be merged, since the job finished successfully, but I think requires some extra tuning (working on it right now):

pomadchin · 2019-05-30T16:38:35Z

@moradology is 98f3656 tested on EMR?

Okay, I got a confirmation from Nathan that all is good.

pomadchin added the needs-cq label Apr 26, 2019

pomadchin changed the title ~~Use AWS SDK v2 [WIP]~~ [WIP] Use AWS SDK v2 Apr 26, 2019

pomadchin added this to the 3.0 milestone May 2, 2019

echeipesh added the in progress label May 3, 2019

echeipesh changed the title ~~[WIP] Use AWS SDK v2~~ AWS SDK v2 May 3, 2019

moradology force-pushed the feature/aws-sdk-v2 branch 2 times, most recently from 837febf to b0551bc Compare May 6, 2019 20:27

echeipesh reviewed May 14, 2019

View reviewed changes

Comment thread s3/src/main/scala/geotrellis/spark/io/s3/package.scala Outdated

echeipesh mentioned this pull request May 14, 2019

Default configuration S3Client #2923

Open

pomadchin reviewed May 15, 2019

View reviewed changes

moradology added 16 commits May 20, 2019 10:01

Use new request structure

b47f257

Init trying to avoid s3client wrapper

2b3d9ae

now compiling

bb7cd1c

Copy AmazonS3Uri from aws sdk v1

4e6688f

Use AmazonS3Uri across s3 project

dab10a8

Add changelog update

d8600fe

Avoid static methods for serialization

046fbe6

All tests passing

590746d

All tests pass

3781211

Run minio in travis testing

777cd4d

Rename getS3Client to getClient

e453f65

Directly specify port for s3 container

c54f9e6

Avoid port 9000 for travis

63a2777

Remove references to s3testkit

500e19a

Update changelog

c7525dc

Pin minio to version before bug

f1ed04d

moradology force-pushed the feature/aws-sdk-v2 branch from 9345a1d to f1ed04d Compare May 20, 2019 14:01

pomadchin self-assigned this May 24, 2019

echeipesh removed the in progress label May 28, 2019

moradology added 2 commits May 29, 2019 09:44

Use geotrellis namespace for aws uri

3ee2125

Use geotrellis namespace for s3uri

8b6468f

moradology force-pushed the feature/aws-sdk-v2 branch from a689ceb to 8b6468f Compare May 29, 2019 14:30

Use aws sdk retrial methods

98f3656

pomadchin removed the needs-cq label May 30, 2019

pomadchin merged commit ce57d8a into locationtech:master May 30, 2019

jbouffard mentioned this pull request Jun 7, 2019

Add Documentation on How to Run the S3 Tests Locally #2974

Open

pomadchin mentioned this pull request Jul 10, 2022

Implement Our Own AWSCredentialsProvider #2708

Closed

Navigation Menu

Repository navigation

AWS SDK v2 - #2911

AWS SDK v2#2911
pomadchin merged 19 commits into
locationtech:masterfrom
moradology:feature/aws-sdk-v2

moradology commented Apr 25, 2019 •

edited by echeipesh

Loading

Uh oh!

Uh oh!

pomadchin May 15, 2019

Uh oh!

moradology commented May 21, 2019

Uh oh!

pomadchin commented May 24, 2019 •

edited

Loading

Uh oh!

pomadchin commented May 24, 2019

Uh oh!

pomadchin commented May 24, 2019

Uh oh!

pomadchin commented May 30, 2019 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

moradology commented Apr 25, 2019 • edited by echeipesh Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Checklist

Notes

Uh oh!

Uh oh!

pomadchin May 15, 2019

Choose a reason for hiding this comment

Uh oh!

moradology commented May 21, 2019

Uh oh!

pomadchin commented May 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pomadchin commented May 24, 2019

Uh oh!

pomadchin commented May 24, 2019

Uh oh!

pomadchin commented May 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

moradology commented Apr 25, 2019 •

edited by echeipesh

Loading

pomadchin commented May 24, 2019 •

edited

Loading

pomadchin commented May 30, 2019 •

edited

Loading