Skip to content

S3RangeReader totalLength Improvment - #3025

Merged
pomadchin merged 4 commits into
locationtech:masterfrom
jbouffard:improvement/s3-header
Jul 19, 2019
Merged

S3RangeReader totalLength Improvment#3025
pomadchin merged 4 commits into
locationtech:masterfrom
jbouffard:improvement/s3-header

Conversation

@jbouffard

@jbouffard jbouffard commented Jul 16, 2019 •
edited by pomadchin
Loading

Copy link
Copy Markdown
Contributor

Overview

This PR changes how the totalLength value of S3RangeReader is computed. It will now use a GetObjectResponse instead of a GetHeaderResponse. This will allow users to read these files even if they don't have permission to access its HEADER.

Checklist

  • docs/CHANGELOG.rst updated, if necessary
  • Check it on a real S3 bucket with permission limits

Closes #2889

@jbouffard

jbouffard commented Jul 16, 2019

Copy link
Copy Markdown
Contributor Author

I'm not really sure there are any unit tests that can be created for this PR. It might be good to test this branch on a real world example.

jbouffard
jbouffard commented Jul 16, 2019
View reviewed changes
Comment thread s3/src/main/scala/geotrellis/store/s3/util/S3RangeReader.scala Outdated
echeipesh
echeipesh reviewed Jul 17, 2019
View reviewed changes
Comment thread s3/src/main/scala/geotrellis/store/s3/util/S3RangeReader.scala Outdated
echeipesh
echeipesh reviewed Jul 17, 2019
View reviewed changes
Comment thread s3/src/main/scala/geotrellis/store/s3/util/S3RangeReader.scala
@jbouffard

jbouffard commented Jul 19, 2019

Copy link
Copy Markdown
Contributor Author

@echeipesh @pomadchin After mentioning my results yesterday, @moradology looked into it as well and found that using a responseStream worked fine for finding the contentLength. I ran the code again this morning and observed similar speeds as Nathan. I'm not sure what happened yesterday 😕 As everything was so much slower. Regardless, this PR should be good to merge now.

pomadchin
pomadchin approved these changes Jul 19, 2019
View reviewed changes

@pomadchin pomadchin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@pomadchin
pomadchin merged commit ef382c9 into locationtech:master Jul 19, 2019
@echeipesh echeipesh mentioned this pull request Sep 23, 2019
Closed
@echeipesh echeipesh mentioned this pull request May 4, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@echeipesh echeipesh echeipesh left review comments
@pomadchin pomadchin pomadchin approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

S3RangeReader will always produce HEAD request

3 participants

@jbouffard @echeipesh @pomadchin