I'm trying to upload files to Google Drive using Google APIs Client Library for JavaScript and resumable upload type.
I authenticate and get the upload URI successfully, but I ran into problems while sending the actual data. If the file contains only ASCII characters, the file is sent successfully to Drive, but in case of special characters (åäö) or binary file (such as PNG) the file gets corrupted. My guess would be that somewhere in the process the file is encoded to unicode in client side.
If I use "btoa()" to encode the raw data to base64 and add header "Content-Encoding: base64" to the data sending request, the file uploads fine. Using this method however increases the overhead for 33%, which is quite a lot when the planned upload size of files is 100MB to 1GB.
Here are some code examples:
Getting the resumable upload URI:
// Authentication is already done
var request = gapi.client.request({
"path": DRIVE_API_PATH, // "/upload/drive/v2/files"
"method": "POST",
"params": {
"uploadType": "resumable"
},
"headers": {
"X-Upload-Content-Type": self.file.type,
//"X-Upload-Content-Length": self.file.size
// If this is uncommented, the upload fails because the file size is
// different (corrupted file). Manually setting to the corrupted file
// size doesn't give 400 Bad Request.
},
"body": {
// self.file is the file object from <input type="file">
"title": self.file.name,
"mimeType": self.file.type,
"Content-Lenght": self.file.size,
}
});
Sending the whole file in one go:
// I read the file using FileReader and readAsBinaryString
// body is the reader.result (or btoa(reader.result))
// and this code is ran after the file has been read
var request = gapi.client.request({
"path": self.resumableUrl, // URI got from previous request
"method": "PUT",
"headers": {
//"Content-Encoding": "base64", // Uploading with base64 works
"Content-Type": self.file.type
},
"body": body
});
Am I missing something? Is it possible to upload file in binary stream? I am new to uploading files in HTML and Javascript and I haven't found any examples using Google Javascript library with resumable upload. There is similar question in SO with no answers.
Blob types are a hot topic for XMLHttpRequest implementations and they are not truly mature. I'd recommend you to stick with base64 encoding. Google's JavaScript client lib doesn't support resumable uploads because it's very unlikely that a client side browser app uploads very large files directly to Google Drive.
What works
To upload a binary blob, use github/googleapi's cors-upload-sample or use my gist fork, UploaderForGoogleDrive, which will grab access_token out of the gapi client for you.
Here is an ugly mixture of Promise and callback code that works for me. As a prerequisite, gapi,UploaderForGoogleDrive, JSZip need to be loaded via <script> tags. The snippet also omits gapi initialization and the API secrets, which are also necessary.
function bigCSV(){ // makes a string for a 300k row CSV file
const rows = new Array(300*1000).fill('').map((v,j)=>{
return [j,2*j,j*j,Math.random(),Math.random()].join(',');
});
return rows.join("\n");
}
function bigZip(){ // makes a ZIP file blob, about 8MB
const zip = new window.JSZip();
zip.folder("A").file("big.csv", bigCSV());
return zip.generateAsync({type:"blob", compression:"DEFLATE"});
// returns Promise<blob>
}
function upload2(zipcontent){
'use strict';
const parent = 'root';
const spaces = 'drive';
const metadata = {
name: 'testUpload2H.zip',
mimeType: 'application/zip',
parents: [parent]
};
const uploader = new window.UploaderForGoogleDrive({
file: zipcontent,
metadata: metadata,
params: {
spaces,
fields: 'id,name,mimeType,md5Checksum,size'
},
onProgress: function(x){
console.log("upload progress:",Math.floor(100*x.loaded/x.total));
},
onComplete: function(x){
if (typeof(x)==='string') x = JSON.parse(x);
// do something with the file metadata in x
console.log("upload complete: ");
},
onError: function(e){ console.log("upload error: ",e); }
});
uploader.upload();
}
function uploadZipFile(){
'use strict';
(bigZip()
.then(upload2)
);
}
What doesn't work
As of Nov 2017, uploading a binary blob with the gapi.client.request call is not going to work, because of an issue where gapi removes the PUT payload
I've also tried using base64 with gapi, which works. but deposits base64 files, not true binaries; and the fetch API in cors mode, which half-worked but produced CORS-related errors and response hiding, at least for me.
Related
The API call to the server is returning a zip file with Content-Disposition in format attachment, <filename>
I am using FileSaver's saveAs to save the file.
this.ajax.raw(requestUrl, {
dataType: 'binary',
xhr: () => {
const myXhr = $.ajaxSettings.xhr()
myXhr.responseType = 'blob'
return myXhr
}
}).then((response) => {
this.downloadSuccess(response, minTimeString, maxTimeString, downloadCompletedMessage)
}).catch((e) => {
this.downloadError(e)
})
downloadSuccess (response, minTime, maxTime, downloadCompletedMessage) {
const filename = (response.jqXHR.getResponseHeader('Content-Disposition').split('"')[1])
saveAs(response.payload, filename, 'application/zip')
This works fine for small files but fails if the file is more than 2Gb (The file is downloaded successfully but the saved file is of 1Kb only).
During my research, I saw that browser can download the file without FileSaver if the response has Content-Disposition which is true in my case. But I am not able to figure out how.
Do I need to use request differently?
From docs:
Content-Disposition attachment header is the best preferred way to
download files from the browser. It has better cross browser
compatibility, won't have any memory limit and it doesn't require any
JavaScript.
You don't need ajax request to download the file. Only ensure that server add Content-Disposition header and provide a link to download.
If you can also use the anchor download attribute from HTML5.
Causes the browser to treat the linked URL as a download.
const link = document.createElement('a');
link.href = '/xyz/abc.pdf';
link.download = "file.pdf";
link.dispatchEvent(new MouseEvent('click'));
I'm trying to upload an arbitrary list of files:
for (let i=0;i<files.length;i++) {
let fileTypeToUse = files[i].type;
fetchWithRetry(url, 1000, 2, {
method: 'POST',
credentials: 'same-origin',
headers: {
'Content-Type': fileTypeToUse
},
body: files[i],
}
}
This works for most file types (including images) by taking the bytes and sending them in the body of the request. But when I try to upload audio of type "audio/mpeg" my server receives a file which is about 60% larger than I expected.
I initially assumed this meant the file was being base64 encoded by the browser, so I tried to decode the file. Unfortunately, this hypothesis seemed to be incorrect. I received a decoding error on the server: base64.StdEncoding.Decode: illegal base64 data at input byte 3
For reference, here is a screenshot of the files object I am trying to upload:
And here is the oversized object being sent by the browser to the server:
Related issue: https://stackoverflow.com/a/40826943/12713117 Unfortunately they are encouraging people to upload the file with form/multipart instead of directly uploading the file.
I have 2 questions. First, why is the uploaded object larger than the file accessible in javascript? Second, how do I know if an arbitrary file will be sent as-is (which is the case with images) or if it will be encoded and need decoding on the server?
Fetch With Retry Code
function fetchWithRetry(url, delay, tries, fetchOptions = {}) {
return fetch(url, fetchOptions).catch((err) => {
let triesLeft = tries - 1;
if (triesLeft == null || triesLeft < 1) {
throw err;
}
return wait(delay).then(() => fetchWithRetry(url, delay * 2, triesLeft, fetchOptions));
});
}
On server there is url with files stored.
Url is http://website.com/abc
I do $http get on this url.
$http.get(url, { responseType: "arraybuffer" });
I want to create Blob object from this. I am sure that object is of type png, because it had this extension before upload and it shows properly.
new Blob(result.data, {type: "image/png"});
I get message:
Failed to construct 'Blob': The 1st argument is neither an array, nor does it have indexed properties.
Response from server http://website.com/abc GET in developer console looks like:
ÿØÿàJFIFÿþ;CREATOR: gd-jpeg v1.0 (using IJG JPEG v62), quality = 60
ÿÛC
' .)10.)-,3:J>36F7,-#WAFLNRSR2>ZaZP`JQROÿÛC&&O5-5OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOÿÂÒ"ÿÄÿÄÿÚ æå2ag\Ý#úDê3[Zdfc5±Ô¢¬æ)K<`¤l2¸ÊánYR±aA`Í%RÈfbz!¤tÞÑ-µd7ZªÀ[hz¨f¥ >©cAV¬{3á R³F0 W>~c³"ðÈìøÖ²ÇBÙ³±+
½ò9tµ°õ
I tried to set Blob type to application/octet-stream and also do $http.get without specified responseType.
How can I create a proper Blob file?
I need Blob File, to create File object which is an entry data to implemented logic to display files in slideshow with modals. I have implemented logic for files of type File, which were created by input forms - implementation was done for a need before uploading to server. Now it turns out that server doesn't return same files to me, but return only urls of files, which created an idea to convert from url to File in order to dont repeat in that logic.
Try
$http.get(url, { responseType: "blob" });
or
// missing `[]` at `js` at Question
new Blob([result.data], {type: "image/png"});
Note XMLHttpRequest responseType could also be set to "blob", see How to build PDF file from binary string returned from a web-service using javascript
I want to download a excel file from my angularJs code. Where i made a http post request to Java Rest API and returned the file with header
"Content-Disposition" : "attachment; filename=\"new_excel_file.xlsx\""
Java Code
#Post
#Path("/excel/trekResult")
#Produces("application/vnd.ms-excel")
public Response getResultsReport(#HeaderParam(Constants.ID) Long userId, #QueryParam(Constants.COMPANY_TREK_ID) Integer companyTrekId) {
String CONTENT_DESPOSITION = "Content-Disposition";
String CONTENT_ATTACHEMENT = "attachment; filename=\"new_excel_file.xlsx\"";
//Generates a excel file in local file system
File excelFile = misHelper.writeToFile(workBook, mis, userId, "trek-results");
return Response.ok().entity((Object)excelFile).
header(CONTENT_DESPOSITION, CONTENT_ATTACHEMENT).build();
}
On Javascript Side
myService.exportResult($scope.companyTrek.id).then(function(result) {
if(result !== undefined || result !== '') {
var blob = new Blob([result], {
type: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
});
var objectUrl = URL.createObjectURL(blob);
saveAs(blob, 'Trek-Results-'+fetchCurrentDate()+ '.xlsx');
}
}
Used FileSaver.js to save file.
The output file is [Object, Object]
Tested The locally generated file.
Here is a similar question for reference that didn't help me.
receive an excel file as response in javascript from a Rest service
I just noticed the Mime types are different on Java server vs Angular client.
This link shows the different MIME types related to spreadsheets.
Try making them consistent and seeing if that fixed it.
There was also this way without mishelper.
I'm trying to upload an image to a web server .aspx page; the server side uses this mechanism to recognize the file:
HttpPostedFile file = Request.Files[0];
I can't change the server code because there are other clients.
Working Javascript/Angular code that can upload from an HTML file input control:
var fileInput = document.getElementById('the-file');
var formData = new FormData();
formData.append('file', fileInput.files[0]);
(etc)
$http.post(uploadUrl, formData, {
headers: { 'Content-Type': undefined },
transformRequest: angular.identity })
All well and good. What I need to figure out, though, is how to upload content from an image that the user has cropped on the client side (and thus the cropped image is no longer affiliated with an HTML file input control). It's in a Canvas, and I know how to get it as a DataUri or an arraybuffer or even a byte array. I just can't figure out how to upload it to this server endpoint so it will be recognized on the server side as a posted file. Is this possible? I don't know how to properly replace the following line:
formData.append('file', fileInput.files[0]);