S3RangeReader will always produce HEAD request

In order to produce object metadata and its overall size the underlying S3 client will issue a HEAD request:

geotrellis/s3/src/main/scala/geotrellis/spark/io/s3/util/S3RangeReader.scala

Lines 39 to 42 in f86ef9e

    
           val metadata: ObjectMetadata = 
        
             client.getObjectMetadata(request.getBucketName, request.getKey) 
        
           val totalLength: Long = metadata.getContentLength

This is problematic for two reasons:

Sometimes the object ACT will not allow HEAD requests, resulting in error 400
In most use cases totalLength is not used and is not important.

To expand on the second point, the GeoTrellis GeoTiffReader will read from the start of the file until it consumes the full header. At that point it will use the offset information from the GeoTiff header to read the data segments.

Note that HttpRangeReader already has logic for this case:

geotrellis/spark/src/main/scala/geotrellis/spark/io/http/util/HttpRangeReader.scala

Lines 38 to 64 in f86ef9e

    
             val totalLength: Long = { 
        
               val headers = if(useHeadRequest) { 
        
                 request.method("HEAD").asString 
        
               } else { 
        
                 request.method("GET").execute { is => "" } 
        
               } 
        
               val contentLength = headers 
        
                   .header("Content-Length") 
        
                   .flatMap({ cl => Try(cl.toLong).toOption }) match { 
        
                     case Some(num) => num 
        
                     case None => -1L 
        
                   } 
        
               /** 
        
                * "The Accept-Ranges response HTTP header is a marker used by the server 
        
                *  to advertise its support of partial requests. The value of this field 
        
                *  indicates the unit that can be used to define a range." 
        
                * https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Ranges 
        
                */ 
        
               require(headers.header("Accept-Ranges") == Some("bytes"), 
        
                 "Server doesn't support ranged byte reads") 
        
               require(contentLength > 0, 
        
                 "Server didn't provide (required) \"Content-Length\" headers, unable to do range-based read") 
        
               contentLength 
        
             }

At a minimum the totalSize should be turned into lazy val which would avoid the issue in most cases. S3RangeReader should also have a constructor flag to avoid head requests. This should cause re-evaluation of the RangeReader interface, maybe we don't need that method at all?

RasterFrames has/had a workaround (in the HTTP case) using a copy of HttpRangeReader:

https://github.com/locationtech/rasterframes/blob/9ced1ec19ead0a9225fb8349b7c0b5b233b79df5/core/src/main/scala/astraea/spark/rasterframes/ref/HttpRangeReader.scala#L42-L49

@echeipesh

@echeipesh I'm a little confused as to how totalLength is rarely used inside S3RangeReader. It looks like it's always checked against when doing range reading.

@jbouffard

@jbouffard it is only used in clipToSize and readAll. totalLength is already a lazy val in S3RangeReader.

I think the issue is to check can we do smth with this HeadRequest usage since it is not always accepted by the S3 bucket policies. If yes - the same constructor can be used as for the HttpRangeReader

@jbouffard

@jbouffard Thats right, its part of that internal function but its rarely/ever used as a public method of the interface. The fact that we use it in clipToSize can be and should be avoided. The issue mentions HttpRangeReader dealing with the same issue so its worth while to take a look at how it's handled there.

Navigation Menu

Repository navigation

S3RangeReader will always produce HEAD request #2889

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	val metadata: ObjectMetadata =
	client.getObjectMetadata(request.getBucketName, request.getKey)

	val totalLength: Long = metadata.getContentLength

	val totalLength: Long = {
	val headers = if(useHeadRequest) {
	request.method("HEAD").asString
	} else {
	request.method("GET").execute { is => "" }
	}
	val contentLength = headers
	.header("Content-Length")
	.flatMap({ cl => Try(cl.toLong).toOption }) match {
	case Some(num) => num
	case None => -1L
	}

	/**
	* "The Accept-Ranges response HTTP header is a marker used by the server
	* to advertise its support of partial requests. The value of this field
	* indicates the unit that can be used to define a range."
	* https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Ranges
	*/
	require(headers.header("Accept-Ranges") == Some("bytes"),
	"Server doesn't support ranged byte reads")

	require(contentLength > 0,
	"Server didn't provide (required) \"Content-Length\" headers, unable to do range-based read")

	contentLength
	}

S3RangeReader will always produce HEAD request #2889

Description

Activity

metasim commented on Apr 3, 2019

jbouffard commented on Jul 16, 2019

pomadchin commented on Jul 16, 2019

echeipesh commented on Jul 16, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions