Figure out how to do browser-based uploading, and document it well.

@Capstan

So there's this concept of "browser-based uploads" which basically is the following process:

Client sends the server an (untrusted) policy document (basically says "I'm going to upload file which is 10mb and has metadata = {...}).
Server gets this request, validates it, and if it likes it, sends back a signature (basically says "SGTM, here's my stamp of approval").
Client is able to upload exactly that data directly to Cloud Storage, it's "credentials" for this single request being the policy document (and the signature).

So... a couple of questions:

Can we even use gcloud-node in the client-side of a web-app?
If so, we need a simple way of creating typical policy documents (as well as weird ones).
Then we need a way of generating a signature (server-side) for a policy document.
Note that we're not wiring together any RPCs or anything. Just providing a method that takes a policy and spits back a signature for the user to serialize and send around however they want.
Then we need a way of calling upload (or similar) using the policy document and signature.

The code I'm expecting here would look like...

Server-side:

bucket.signPolicy(policy, function(signature) {
  // The signature can be serialized or whatever else we want now.
});

Client-side:

var policy = bucket.policy({ /* Lots of configuration here... */ });
$.post({ /* Send the policy up to the method above */ }, function(response) {
  // Create an HTML form (?) that has a field called signature = response.data ?
});

/cc @Capstan

@jgeewax

@jgeewax @stellanhaglund will this solution work?

Docs: https://cloud.google.com/storage/docs/access-control?hl=en#signing-resumable
Issue: Authorization: Bearer your_auth_token (how to get auth_token) #818
Solution: Authorization: Bearer your_auth_token (how to get auth_token) #818 (comment)
(this can be simplified with https://googlecloudplatform.github.io/gcloud-node/#/docs/v0.21.0/storage/file?method=createResumableUpload)

I made a couple of examples here showing two different ways to handle uploads. https://github.com/stephenplusplus/gcloud-upload-example

Thanks alot!!

I just added that but now I'm trying to download a file using

app.get('/file/:imageId', function(req, res){

var remoteFile = bucket.file(req.params.imageId);
var localFilename = 'uploads/' + req.params.imageId;

var src = remoteFile.createReadStream()

src
.on('error', function(err) {})
.on('response', function(response) {
// Server connected and responded with the specified status and headers.
})
.on('end', function() {
// The file is fully downloaded.
})
// .pipe(fs.createWriteStream(localFilename));

res.attachment(localFilename);
src.pipe(res);

})

but for larger files I just get about 1.2 mb downloaded..

2015-09-17 1:27 GMT+02:00 Stephen Sawchuk [email protected]:

I made a couple of examples here showing two different ways to handle
uploads. https://github.com/stephenplusplus/gcloud-upload-example

—
Reply to this email directly or view it on GitHub
#641 (comment)
.

Which version of gcloud are you running?

"gcloud": "^0.21.0",

2015-09-17 15:28 GMT+02:00 Stephen Sawchuk [email protected]:

Which version of gcloud are you running?

—
Reply to this email directly or view it on GitHub
#641 (comment)
.

Hmm, I haven't been able to reproduce. How large are the files you're having trouble with? I've tried with a 10MB file without issues.

I added onto the example with some options of how to download files: https://github.com/stephenplusplus/gcloud-upload-example/blob/1d6cbecb31aa2dcd98d1c8177e6c474b80b5d1f5/server.js#L89

about 300mb, and the files will probably be up to some gb's

2015-09-17 16:16 GMT+02:00 Stephen Sawchuk [email protected]:

Hmm, I haven't been able to reproduce. How large are the files you're
having trouble with? I've tried with a 10MB file without issues.

I added onto the example with some options of how to download files:
https://github.com/stephenplusplus/gcloud-upload-example/blob/1d6cbecb31aa2dcd98d1c8177e6c474b80b5d1f5/server.js#L89

—
Reply to this email directly or view it on GitHub
#641 (comment)
.

Would you mind opening a new issue about this? Also, please test that it still doesn't work isolated from express, e.g.:

remoteFile.createReadStream().pipe(fs.createWriteStream('./backup-file'))

I think we've got this one covered with file#createResumableUpload().

Related, I just stumbled upon this project over the weekend: https://github.com/makeomatic/ms-files-gce

If there's more we can do, let me know!

chore(deps): update dependency gts to v3 (#641)

chore: remove duplicate mocha config (#641)

build: update auto approve to v2, remove release autoapproving (#1432) (

chore: remove duplicate mocha config (#641)

build: update auto approve to v2, remove release autoapproving (#1432) (

chore(deps): update dependency tmp to ^0.1.0 (#641)

build: update auto approve to v2, remove release autoapproving (#1432) (

chore(deps): update dependency @types/async to v3 (#641)

build(node): add KOKORO_BUILD_ARTIFACTS_SUBDIR to env (#641)

fix: release uuid rollback (#641)

feat: release 6.2.0 (#641)

chore(deps): update dependency tmp to ^0.1.0 (googleapis#641)

fix: improve typescript types for Table class (#641)

fix: improve typescript types for Table class (googleapis#641)

Navigation Menu

Repository navigation

Figure out how to do browser-based uploading, and document it well. #641

18 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Figure out how to do browser-based uploading, and document it well. #641

Description

Activity

stephenplusplus commented on Sep 16, 2015

stephenplusplus commented on Sep 16, 2015

stellanhaglund commented on Sep 17, 2015

stephenplusplus commented on Sep 17, 2015

stellanhaglund commented on Sep 17, 2015

stephenplusplus commented on Sep 17, 2015

stellanhaglund commented on Sep 17, 2015

stephenplusplus commented on Sep 17, 2015

stephenplusplus commented on Nov 23, 2015

18 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions