Client Library should stop sending x-goog-api-client, which otherwise needs to be included in a signed URL #823

Activity

1. product-auto-label

   added

   api: storageIssues related to the googleapis/java-storage API.Issues related to the googleapis/java-storage API.

   on May 11, 2021
2. cojenco

   self-assigned this

   on May 12, 2021
3. yoshi-automation

   added

   triage meI really want to be triaged.I really want to be triaged.

   on May 14, 2021
4. cojenco

   added

   status: investigatingThe issue is under investigation, which is determined to be non-trivial.The issue is under investigation, which is determined to be non-trivial.

   type: questionRequest for information or clarification. Not an issue.Request for information or clarification. Not an issue.

   and removed

   triage meI really want to be triaged.I really want to be triaged.

   status: investigatingThe issue is under investigation, which is determined to be non-trivial.The issue is under investigation, which is determined to be non-trivial.

   on May 14, 2021
5. BenWhitehead

   assigned

   BenWhitehead

   and unassigned

   cojenco

   on May 17, 2021
6. BenWhitehead commented on May 19, 2021

   

   BenWhitehead

   on May 19, 2021

   Collaborator

   More actions

   Hi @blackhogz,

   I've spent some time trying to reproduce the error (starting from the linked Javadoc snippet) that you're reporting but I haven't been able to.

   Can you provide any more information on the environment/configuration you're using when you run into this error?

   1. What version(s) of com.google.cloud:google-cloud-storage are you using?
   2. When calling storage.writer(signedURL) is it called from the same client used to generate signedURL?
   3. If signedURL is being used somewhere other than from the client that generated it can you provide some details of where it is being used from?
7. natheihei commented on May 20, 2021

   

   natheihei

   on May 20, 2021

   Author

   More actions

   Hi @BenWhitehead and thanks a lot for helping with the issue.

   Here's the main file that demonstrates the issue (I've replaced some actual project/bucket name with placeholder), and the gradle settings.

   // UploadMain.java
   package test;
   
   import java.io.IOException;
   import java.net.URL;
   import java.nio.ByteBuffer;
   import java.util.HashMap;
   import java.util.Map;
   import java.util.concurrent.TimeUnit;
   
   import com.google.cloud.WriteChannel;
   import com.google.cloud.storage.BlobId;
   import com.google.cloud.storage.BlobInfo;
   import com.google.cloud.storage.HttpMethod;
   import com.google.cloud.storage.Storage;
   import com.google.cloud.storage.StorageOptions;
   
   import static java.nio.charset.StandardCharsets.UTF_8;
   
   public class UploadMain {
   
       public static void main(String[] args) {
           // Generating the Signed URL utilizing *authenticated* storage service, to simulate what will happen on a backend
           // server.
           Storage authenticatedStorage = StorageOptions.getDefaultInstance()
                   .toBuilder()
                   .setProjectId("project-id-placeholder").build().getService();
           String bucketName = "bucket-name-placeholder";
           String blobName = "blob-name-placeholder";
           BlobId blobId = BlobId.of(bucketName, blobName);
           BlobInfo blobInfo = BlobInfo.newBuilder(blobId).setContentType("application/octet-stream").build();
           Map<String, String> extensionHeaders = new HashMap<>();
           extensionHeaders.put("x-goog-resumable", "start");
           //
           // This line below is critical and is what this issue (https://github.com/googleapis/java-storage/issues/823)
           // is about. Without it, the upload later with the unauthenticated storage will fail.
           //
           // extensionHeaders.put("x-goog-api-client", "gl-java/1.8.0_292 gccl/1.113.16 gax/1.63.0");
           URL signedURL = authenticatedStorage.signUrl(blobInfo,
                   15,
                   TimeUnit.MINUTES,
                   Storage.SignUrlOption.httpMethod(HttpMethod.POST),
                   Storage.SignUrlOption.withExtHeaders(extensionHeaders)
           );
           System.out.println("The signed url is " + signedURL);
   
           // Using the Signed URL with another *unauthenticated* storage, to simulate what will happen on a client.
           Storage unauthenticatedStorage = StorageOptions.getUnauthenticatedInstance().getService();
           byte[] content = "Hello, World!".getBytes(UTF_8);
           try (WriteChannel writer = unauthenticatedStorage.writer(signedURL)) {
               writer.write(ByteBuffer.wrap(content, 0, content.length));
           } catch (IOException e) {
               e.printStackTrace();
           }
       }
   }
   

   and

   # build.gradle
   plugins {
       id 'java'
   }
   
   group 'org.example'
   version '1.0-SNAPSHOT'
   
   repositories {
       mavenCentral()
   }
   
   dependencies {
       compile 'com.google.cloud:google-cloud-storage:1.113.16'
       testCompile group: 'junit', name: 'junit', version: '4.12'
   }
   

   And to answer your questions:

   1. What version(s) of com.google.cloud:google-cloud-storage are you using?

   'com.google.cloud:google-cloud-storage:1.113.16'

   2. When calling storage.writer(signedURL) is it called from the same client used to generate signedURL?

   It is called from another client (an unauthenticated Storage instance) to simulate the real use case, where the server with a GCP service account credential signs an URL and the client, without any credential, utilizes the signed URL to access cloud storage service.

   3. If signedURL is being used somewhere other than from the client that generated it can you provide some details of where it is being used from?

   Yes, as you can see from the attached UploadMain.java, it is used by another unauthenticated Storage instance, different from the client that generates the signed URL. In fact, I feel the example code given in the comment may be misleading and not representing the real use case scenario of signed URL, where the consumer of the URL and the producer are different Storage instances.

   Thanks again, let me know how I can help!
8. BenWhitehead commented on May 21, 2021

   

   BenWhitehead

   on May 21, 2021

   Collaborator

   More actions

   Thank you for helping out with a reproduction.

   I agree with you, tying a signed url to a specific client cuts off the use case you've described. I'll have to reach out to some folks and get their opinion on the criticality of x-goog-api-client being part of a signed url and if it'll be okay for us to mark it as excluded similar to x-goog-encryption-key.

   👍React with 👍1Reacted by natheihei
9. natheihei commented on May 22, 2021

   

   natheihei

   on May 22, 2021

   Author

   More actions

   Thanks @BenWhitehead . Keep me posted! Let me know if anything I can help :D
10. natheihei commented on Jul 7, 2021

    

    natheihei

    on Jul 7, 2021

    Author

    More actions

    Kindly ping? Any updates?
11. BenWhitehead commented on Jul 8, 2021

    

    BenWhitehead

    on Jul 8, 2021

    Collaborator

    More actions

    @blackhogz We've got a fix ready in #915, apologies for the delay there were a number of code paths and cases we had to track down to settle on this fix.

    We're also investigating adding a dedicated Signed url option to include the x-goog-resumable: start rather than having to provide it via extension headers.
12. JesseLovelace

    closed this as completedin #915on Jul 12, 2021