Repository navigation
Problems with metadata-only requests and media MIME types #229
Description
Activity
Hi @AndrewMagliozzi, thanks for the report. I'll look into this.
We should set
isMediaonly if media is given or create multipart if media resource exists. It looks like a quick fix.Hi @rakyll that's a great start, but it's only part of the problem as I see it.
The URL would also need to be modified, because 'upload' is hard-coded in the api endpoint in places like this: https://github.com/google/google-api-nodejs-client/blob/master/apis/gmail/v1.js#L59
This is particularly relevant for api calls like this one where there are two end points (one for upload and one for metadata-only): https://developers.google.com/gmail/api/v1/reference/users/drafts/create
Thanks for your fast replies!
Yep. We will have to take into consideration both URLs. I'm working on a fix.
"/upload"is just appended to the end of every URL so we can just trim it off for the short term.It's prepended to the path. I'd not rely on hard-coded find and replace though. Discovery doc provides paths for media and non-media endpoints.
There may still be an issue with adjusting the mimetype as well. createAPIRequest may need some adjusting too.
This is where the mimeType gets set: https://github.com/google/google-api-nodejs-client/blob/master/lib/apirequest.js#L121
Unfortunately, the GMail API returns this error when I try to specify a different mimetype in the options object:
Unknown field name: mimeTypeYou may want to address the mimeType param like you do here with some of the other parts of the options object: https://github.com/google/google-api-nodejs-client/blob/master/lib/apirequest.js#L78-L84
I've been looking at the discovery API, and I don't see where it defines the distinction between metadata-only and upload. As far as I can tell, API methods can have either 1. only non-upload URLS, 2. both non-upload and upload URLs, or 3. only upload URLs. Here are examples of these three cases:
- Only non-upload: https://developers.google.com/drive/v2/reference/files/copy. Here is the description I get for this method from the discovery API:
{ "id": "drive.files.copy", "path": "files/{fileId}/copy", "httpMethod": "POST", "description": "Creates a copy of the specified file.", "parameters": { ... }, "parameterOrder": [ "fileId" ], "request": { "$ref": "File" }, "response": { "$ref": "File" }, "scopes": [ "https://www.googleapis.com/auth/drive", "https://www.googleapis.com/auth/drive.appdata", "https://www.googleapis.com/auth/drive.apps.readonly", "https://www.googleapis.com/auth/drive.file" ] }- Upload and non-upload: https://developers.google.com/drive/v2/reference/files/insert.
{ "id": "drive.files.insert", "path": "files", "httpMethod": "POST", "description": "Insert a new file.", "parameters": { ... }, "request": { "$ref": "File" }, "response": { "$ref": "File" }, "scopes": [ "https://www.googleapis.com/auth/drive", "https://www.googleapis.com/auth/drive.appdata", "https://www.googleapis.com/auth/drive.apps.readonly", "https://www.googleapis.com/auth/drive.file" ], "supportsMediaUpload": true, "mediaUpload": { "accept": [ "*/*" ], "maxSize": "5120GB", "protocols": { "simple": { "multipart": true, "path": "/upload/drive/v2/files" }, "resumable": { "multipart": true, "path": "/resumable/upload/drive/v2/files" } } }, "supportsSubscription": true }- Only upload: https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/uploads/uploadData.
{ "id": "analytics.management.uploads.uploadData", "path": "management/accounts/{accountId}/webproperties/{webPropertyId}/customDataSources/{customDataSourceId}/uploads", "httpMethod": "POST", "description": "Upload data for a custom data source.", "parameters": { ... }, "parameterOrder": [ "accountId", "webPropertyId", "customDataSourceId" ], "response": { "$ref": "Upload" }, "scopes": [ "https://www.googleapis.com/auth/analytics", "https://www.googleapis.com/auth/analytics.edit" ], "supportsMediaUpload": true, "mediaUpload": { "accept": [ "application/octet-stream" ], "maxSize": "1GB", "protocols": { "simple": { "multipart": true, "path": "/upload/analytics/v3/management/accounts/{accountId}/webproperties/{webPropertyId}/customDataSources/{customDataSourceId}/uploads" }, "resumable": { "multipart": true, "path": "/resumable/upload/analytics/v3/management/accounts/{accountId}/webproperties/{webPropertyId}/customDataSources/{customDataSourceId}/uploads" } } } }The distinction between the first and the last two is pretty obvious: the last two have supportsMediaUpload: true and a mediaUpload key with accepted mime types, max size, and alternate endpoints. Interestingly, they define separate endpoints for "resumable" uploads, which are not described in any documentation I can find.
However, I don't see a clear distinction between the second and third discovery documents. The second includes supportsSubscription: true while the third does not. It also includes a request key while the third does not. Of these two differences, the request key seems like the most likely candidate as the relevant distinction, but I'm still not convinced. Could there be a method which has both upload and non-upload endpoints and where a request object is never passed? Maybe not, but this all still seems very ambiguous.
This is a great thought. I checked all the cases that have no request object and accept media and most seem to follow the pattern that you're describing except it seems that
fusiontables.table.importRowsis documented in the discovery docs to not accept a request object but in the documentation itself it notes that it accepts both metadata only and media uploads.So because of this, we can only decide what kind of upload to take at runtime based on the parameters you pass in. Unless you can find a better pattern 😄
fusiontables.table.importTablehas the same issue as described too. Seems these are the only two.Internal bug filed (b/16576736) to see if this is unexpected as what you are describing seems to make sense.
So the
mimeTypeissue is a good thing to raise but also seems different because I used the API Explorer to create a draft with only metadata and theContent-Typeof the request wasapplication/jsonbecause metadata is always specified in a JSON format. But if you specify a body to the media, then you also need to specify amimeTypein this case equal tomessage/rfc822Hi @ryanseys you're right, the mimeType is slightly a different issue. Unfortunately, even when I try to specify the proper mimeType ('message/rfc822') in the resource object for a request with media to the GMail api for a function like users.drafts.create, I get this error:
Unknown field name: mimeTypeIf I don't specify mimeType, I get this error:
Media type \'application/octet-stream\' is not supported. Valid media types: [message/rfc822]Currently, I am stumped. Thanks for your help!
@AndrewMagliozzi Yes, we have a possible fix available in #230 that you can try out.
Thanks!
On Sat, Jul 26, 2014 at 2:03 PM, Ryan Seys [email protected] wrote:
@AndrewMagliozzi https://github.com/AndrewMagliozzi Yes, we have a
possible fix available in #230
#230 that you
can try out.—
Reply to this email directly or view it on GitHub
#229 (comment)
.@ryanseys we just tested again.
Here is our attempt with a media object as part of the API call:
gmail.users.drafts.create({ auth: authClient, userId: 'me', media: { mimeType: 'message/rfc822', body: email } }, function(err, response) { if (err) { console.log(err); } else { console.log(response); } });Here is the error that results:
{ errors: [ { domain: 'global', reason: 'badContent', message: 'Unsupported content with type: message/rfc822' } ], code: 400, message: 'Unsupported content with type: message/rfc822' }When I execute the API call without a media object and specify the mimetype:
gmail.users.drafts.create({ auth: authClient, userId: 'me', resource: { mimeType: 'message/rfc822' }, message: { raw: base64encodedmessage } });Here is the error that results:
{ errors: [ { domain: 'global', reason: 'invalid', message: 'Unknown field name: mimeType', locationType: 'other', location: '' } ], code: 400, message: 'Unknown field name: mimeType' }When I execute the API call without a media object only:
gmail.users.drafts.create({ auth: authClient, userId: 'me', message: { raw: base64encodedmessage } });Here is the error that results:
{ errors: [ { domain: 'global', reason: 'backendError', message: 'Backend Error' } ], code: 500, message: 'Backend Error' }Scratch that, SUCCESS!
We adjusted the syntax and added an email delegate as the last parameter in the authClient object (apparently this wasn't "optional"). Once 'me' was sufficiently defined in our API request, this syntax worked for us:
gmail.users.drafts.create({ auth: authClient, userId: 'me', resource: { message: { raw: base64EncodedEmail } } });So the last two won't work because
mimeTypeis only specified inmedia.mimeTypefor that API and yourmessageneeds to go insideresource. I'm guessing the first one isn't working because you didn't specify a resource. Can you try to specifyresource: {}in the first example and see if you have any success? This might be an issue with https://github.com/google/google-api-nodejs-client/blob/master/lib/apirequest.js#L111If by specifying an empty resource object we have success, then I will need to remove the check at line 111 in the previous comment.
As @AndrewMagliozzi said, we got it to work. However, regarding the need to specify an empty resource: I think what you actually need to do when no resource is specified is to not send a multi-part request.
@AndrewMagliozzi @atrigent check out #235 and test with that branch (https://github.com/ryanseys/google-api-nodejs-client/tree/refresh-before). It should not build a multipart for media only requests.
This sounds like a similar issue I am seeing with BigQuery in V1.0, I am trying to insert a new BigQuery job (bigquery.jobs.insert), and updated the formatting of my request to move the body data from the second parameter to adding it as a resource (also using the global auth object, thanks for that one :).
// NEW params = { projectId: "", resource: body }; bigquery.jobs.insert(params, function(err, result) { }); // OLD googleapis.bigquery.jobs.insert(params, body).withAuthClient(oauth2).execute(..);
but I am now seeing this error from BigQuery now:
{ errors: [ { domain: 'global', reason: 'invalid', message: 'Uploads are only allowed with load jobs' } ], code: 400, message: 'Uploads are only allowed with load jobs' }
Looking through the source code bigquery.jobs.insert is being marked as isMedia: true, even if I don't have any media, which I think might be causing the issue.
Just released
1.0.3which addresses this issue.https://github.com/google/google-api-nodejs-client/releases/tag/1.0.3
Thanks again, Ryan. It works great now!
On Wed, Jul 30, 2014 at 7:57 PM, Ryan Seys [email protected] wrote:
—
Reply to this email directly or view it on GitHub
#229 (comment)
.
Thanks for the recent updates to this npm package. It's great, but I've been having some issues with the GMail API since the update to version 1.0.0.
For at least the GMail API and possibly other apis as well, there is little support for metadata-only requests.
Looking at the api code, 'isMedia' is always set for true in a function like users.drafts.create, as seen here: https://github.com/google/google-api-nodejs-client/blob/master/apis/gmail/v1.js#L65. Also, the /uploads URL is always used although some APIs support metadata-only requests with a different URL.
A related issue is that it is impossible to correctly set the media MIME type for multipart uploads, which is all the code currently supports. The GMail users.drafts.create method, for example, requires a media MIME type of message/rfc822. If we make no effort to set this, we get the following API error:
If, however, we add
resource: { mimeType: 'message/rfc822' }to the options object, we get the following API error:This is because the gmail API does not have a mimeType field. Unfortunately, this is where the
createAPIRequestfunction looks for setting the media MIME type. This seems to work for some APIs, such as drive, which actually do have a mimeType field, but it does not work for gmail.Thanks for any help you can provide in resolving this issue.