I am trying to retrieve a file from the user's file system and upload it to AWS S3. However, I have had no success thus far doing so. To be more specific, I have been working with trying to upload images. So far the images won't render properly whenever I upload them. I am really only familiar with uploading images as Blobs, but since SHA256 function can't read a Blob I'm unsure what to do. Below is my code:
var grabFile = new XMLHttpRequest();
grabFile.open("GET", 'https://s3.amazonaws.com/'+bucketName+'/asd.jpeg', true);
grabFile.responseType = "arraybuffer";
grabFile.onload = function( e ) {
var grabbedFile = this.response;
var arrayBufferView = new Uint8Array( this.response );
var blob = new Blob( [ arrayBufferView ], { type: "image/jpeg" } );
var base64data = '';
var reader = new FileReader();
reader.readAsDataURL(blob);
reader.onloadend = function() {
//var readData = reader.result;
var readData = blob;
//readData = readData.split(',').pop();
console.log(readData);
console.log(readData.toString());
var shaString = CryptoJS.SHA256(readData.toString()).toString();
var request = new XMLHttpRequest();
var signingKey = getSigningKey(dateStamp, secretKey, regionName, serviceName);
var headersList = "content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date";
var time = new Date();
time = time.toISOString();
time = time.replace(/:/g, '').replace(/-/g,'');
time = time.substring(0,time.indexOf('.'))+"Z";
var canonString = "PUT\n"+
"/asd5.jpeg\n"+
"\n"+
//"content-encoding:base64\n"+
"content-type:image/jpeg\n"+
"host:"+bucketName+".s3.amazonaws.com\n"+
'x-amz-acl:public-read\n'+
'x-amz-content-sha256:'+shaString+"\n"+
'x-amz-date:'+time+'\n'+
'\n'+
headersList+'\n'+
shaString;
var stringToSign = "AWS4-HMAC-SHA256\n"+
time+"\n"+
dateStamp+"/us-east-1/s3/aws4_request\n"+
CryptoJS.SHA256(canonString);
var authString = CryptoJS.HmacSHA256(stringToSign, signingKey).toString();
var auth = "AWS4-HMAC-SHA256 "+
"Credential="+accessKey+"/"+dateStamp+"/"+regionName+"/"+serviceName+"/aws4_request, "+
"SignedHeaders="+headersList+", "+
"Signature="+authString;
request.open("PUT", "https://"+bucketName+".s3.amazonaws.com/asd5.jpeg", true);
request.setRequestHeader("Authorization", auth);
//request.setRequestHeader("content-encoding", "base64");
request.setRequestHeader("content-type", "image/jpeg");
request.setRequestHeader('x-amz-acl', 'public-read');
request.setRequestHeader("x-amz-content-sha256", shaString);
request.setRequestHeader("x-amz-date", time);
request.send(readData.toString());
console.log(request);
How do I go about doing this? The code above just uploads something that's just a few Bytes because blob.toString() comes out as [Object Blob] and that's what gets uploaded. If I don't toString() it, I get an error from my SHA256 function.
As you can see, I tried reading it as Base64 before uploading it, but that did not resolve my problem either. This problem has been bugging me for close to a week now and I would love to get this solved. I've tried changing content-type, changed the body of what I'm uploading, etc. but nothing has worked.
EDIT: Forgot to mention (even though the title should imply it) but I cannot use the SDK for this. I was using it at one point, and with it I was able to upload images as Blobs. So I know this is possible, it's just that I don't know what crafty thing the SDK is doing to upload it.
EDIT2: I found the solution just in case someone stumbles across this in the future. Try setting replace every place I have shaString with 'UNSIGNED PAYLOAD' and send the blob and it will work! Here's where I found it: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
Related
I am streaming ArrayBuffers from a python server and am trying to interpret each one as an image on the client side with javascript. They are being received as arraybuffers in javascript. However I cant get them to be readable by the image tags src attribute. I have tried generating them into Blob objects then using window.URL.createObjectURL(blob). That hasnt work either.
The blob url looks like this blob:null/e2836074-64b5-4959-8211-da2fc24c35a6 is that wrong?
Does any have any suggestions/know a solution.
Thanks a lot.
var arrayBuffer = new Uint8Array(stream.data);
var blob = new Blob([arrayBuffer], {type: "image/jpeg"});
var urlCreator = window.URL || window.webkitURL;
var imageUrl = urlCreator.createObjectURL( blob );
console.log(imageUrl);
img.src = imageUrl;
array buffer image
If you have any control over things, you should use the responseType of blob on your Javascript call. This will let you use the data you are getting from your server directly instead of attempting to access it via an ArrayBuffer
See the following fiddle for an example: https://jsfiddle.net/ort74gmp/
// Simulate a call to Dropbox or other service that can
// return an image as a blob
var xhr = new XMLHttpRequest();
// Use JSFiddle logo as a sample image to avoid complicating
// this example with cross-domain issues.
xhr.open( "GET", "https://fiddle.jshell.net/img/logo.png", true );
// Ask for the result as an ArrayBuffer.
xhr.responseType = "blob";
xhr.onload = function( e ) {
var blob = this.response;
var reader = new FileReader();
reader.onload = function() {
var dataURL = reader.result;
document.querySelector('#photo').src = dataURL;
}
reader.readAsDataURL(blob);
};
xhr.send();
I realize that the new Mozilla Firefox return allocation size overflow (on FileReader.ReadAsBinaryString()) when the file bigger than 200MB (something like that).
Here's some of my code on test for client web browser:
function upload(fileInputId, fileIndex)
{
var file = document.getElementById(fileInputId).files[fileIndex];
var blob;
var reader = new FileReader();
reader.readAsBinaryString(file);
reader.onloadend = function(evt)
{
xhr = new XMLHttpRequest();
xhr.open("POST", "upload.php", true);
XMLHttpRequest.prototype.mySendAsBinary = function(text){
var data = new ArrayBuffer(text.length);
var ui8a = new Uint8Array(data, 0);
for (var i = 0; i < text.length; i++){
ui8a[i] = (text.charCodeAt(i) & 0xff);
}
if(typeof window.Blob == "function")
{
blob = new Blob([data]);
}else{
var bb = new (window.MozBlobBuilder || window.WebKitBlobBuilder || window.BlobBuilder)();
bb.append(data);
blob = bb.getBlob();
}
this.send(blob);
}
var eventSource = xhr.upload || xhr;
eventSource.addEventListener("progress", function(e) {
var position = e.position || e.loaded;
var total = e.totalSize || e.total;
var percentage = Math.round((position/total)*100);
});
xhr.onreadystatechange = function()
{
if(xhr.readyState == 4)
{
if(xhr.status == 200)
{
console.log("Done");
}else{
console.log("Fail");
}
}
};
xhr.mySendAsBinary(evt.target.result);
};
}
So I tried change it to FileReader.ReadAsArrayBuffer(), the error has not shown up but the data are not the same (as it's not read as binary string).
Did anyone has any solution to solve this problem? Is there any way that we can upload bigger file from JS to Web Server in raw/string other than FileReader implementation?
I read on Mozilla JS Documentation that said:
This feature is non-standard and is not on a standards track. Do not
use it on production sites facing the Web: it will not work for every
user. There may also be large incompatibilities between
implementations and the behavior may change in the future. - Mozilla
If not ReadAsBinaryString, the how to implement ReadAsArrayBuffer or ReadAsText
To send Files to a web-server, you simply don't need js. HTML alone is well able to do this with the <form> element.
Now if you want to go through js, for e.g catch the different ProgressEvents, then you can send directly your File, no need to read it whatsoever on your side.
To do this, you've got two (or three) solutions.
If your server is able to handle PUT requests, you can simply xhr.send(file);.
Otherwise, you'd have to go through a FormData.
// if you really want to go the XHR way
document.forms[0].onsubmit = function handleSubmit(evt) {
if(!window.FormData) { // old browser use the <form>
return;
}
// now we handle the submit through js
evt.preventDefault();
var fD = new FormData(this);
var xhr = new XMLHttpRequest();
xhr.onprogress = function handleProgress(evt){};
xhr.onload = function handleLoad(evt){};
xhr.onerror = function handleError(evt){};
xhr.open(this.method, this.action);
// xhr.send(fD); // won't work in StackSnippet
log(fD, this.method, this.action); // so we just log its content
};
function log(formData, method, action) {
console.log('would have sent');
for(let [key, val] of formData.entries())
console.log(key, val);
console.log('through', method);
console.log('to', action);
}
<!-- this in itself is enough -->
<form method="POST" action="your_server.page">
<input type="file" name="file_upload">
<input type="submit">
</form>
Now, you sent a comment saying that you can't upload Files bigger than 1GB to your server.
This limitation is only due to your server's config, so the best if you want to accept such big files is to configure it correctly.
But if you really want to send your File by chunks, even then don't get off of the Blob interface.
Indeed Blobs have a slice() method, so use it.
document.forms[0].onsubmit = function handleSubmit(evt) {
evt.preventDefault();
var file = this.elements[0].files[0];
var processed = 0;
if(file) {
// var MAX_CHUNK_SIZE = Math.min(file.size, server_max_size);
// for demo we just split in 10 chunks
var MAX_CHUNK_SIZE = file.size > 10 ? (file.size / 10) | 0 : 1;
loadChunk(0);
}
function loadChunk(start) {
var fD = new FormData();
var sliced = file.slice(start, start+MAX_CHUNK_SIZE);
processed += sliced.size; // only for demo
fD.append('file_upload', sliced, file.name);
fD.append('starting_index', start);
if(start + MAX_CHUNK_SIZE >= file.size) {
fD.append('last_chunk', true);
}
var xhr = new XMLHttpRequest();
xhr.open('POST', 'your_server.page');
xhr.onload = function onchunkposted(evt) {
if(start + MAX_CHUNK_SIZE >= file.size) {
console.log('All done. Original file size: %s, total of chunks sizes %s', file.size, processed);
return;
}
loadChunk(start + MAX_CHUNK_SIZE);
};
// xhr.send(fD);
log(fD);
setTimeout(xhr.onload, 200); // fake XHR onload
}
};
function log(formData, method, action) {
console.log('would have sent');
for(let [key, val] of formData.entries())
console.log(key, val);
}
<form method="POST" action="your_server.page">
<input type="file" name="file_upload">
<input type="submit">
</form>
But you absolutely don't need to go through a FileReader for this operation.
Actually the only case where it could make sense to use a FileReader here would be for some Android browsers that don't support passing Blob into a FormData, even though they don't give a single clue about it.
So in this case, you'd have to set up your server to let you know the request was empty, and then only read the File as a dataURI that you would send in-place of the original File.
after a long week of research and sleepless nights, you can't upload binary strings without breaking it, also base64 doesn't work for all files, only images, the journey from the client-side to the server breaks the bytes being sent
Kaiido statement is correct
To send Files to a web-server, you simply don't need js
But that doesn't answer my question. Using the Simple XMLHttpRequest() can upload the file and track those progress as well. But still, it's not it. The direct upload, either from the <form> or using XMLHttpRequest() will need to increase your upload limit in php setting. This method is not convenience for me. How if the client upload file as 4GB? So I need to increase to 4GB. Then next time, client upload file as 6GB, then I have to increase to 6GB.
Using the slice() method is make sense for bigger file as we can send it part by part to server. But this time I am not using it yet.
Here's some of my test the worked as I want. I hope some expert could correct me if I am wrong.
My Upload.js
function upload(fileInputId, fileIndex)
{
var file = document.getElementById(fileInputId).files[fileIndex];
var blob;
var reader = new FileReader();
reader.readAsArrayBuffer(file);
reader.onloadend = function(evt)
{
xhr = new XMLHttpRequest();
xhr.open("POST", "upload.php?name=" + base64_encode(file.name), true);
XMLHttpRequest.prototype.mySendAsBinary = function(text){
var ui8a = new Uint8Array(new Int8Array(text));
if(typeof window.Blob == "function")
{
blob = new Blob([ui8a]);
}else{
var bb = new (window.MozBlobBuilder || window.WebKitBlobBuilder || window.BlobBuilder)();
bb.append(ui8a);
blob = bb.getBlob();
}
this.send(blob);
}
var eventSource = xhr.upload || xhr;
eventSource.addEventListener("progress", function(e) {
var position = e.position || e.loaded;
var total = e.totalSize || e.total;
var percentage = Math.round((position/total)*100);
console.log(percentage);
});
xhr.onreadystatechange = function()
{
if(xhr.readyState == 4)
{
if(xhr.status == 200)
{
console.log("Done");
}else{
console.log("Fail");
}
}
};
xhr.mySendAsBinary(evt.target.result);
};
}
Below is how the PHP server listen to the ArrayBuffer from JS
if(isset($_GET["name"])){
$name = base64_decode($_GET["name"]);
$loc = $name;
$inputHandler = fopen('php://input', "r");
$fileHandler = fopen($loc, "w+");
while(true) {
//$buffer = fgets($inputHandler, 1024);
$buffer = fread($inputHandler, 1000000);
if (strlen($buffer) == 0) {
fclose($inputHandler);
fclose($fileHandler);
return true;
}
//$b = base64_encode($buffer);
fwrite($fileHandler, $buffer);
}
}
The above method works well. The FileReader read the file as ArrayBuffer the upload to server. For me, migrating from ReadAsBinaryString() to ReadAsArrayBuffer() is important and ReadAsArrayBuffer() has some better performance rather than ReadAsBinaryString()
Here's some reason, why some developer relies to FileReader API:
Streaming. Using this method, the file will be stream, so we can avoid setting the php multiple time.
Easy Encrypt. As the file is sending via ArrayBuffer, it is easy for developer to Encrypt the file while upload in progress.
This method also support upload any type of file. I ve done some test and I realize that ReadAsArrayBuffer() method are more faster than ReadAsBinaryString() and direct form upload. You may try it.
Security Notice
The above code is only under test code, to use it in production, you have to consider sending the data in GET or POST under HTTPS.
I'm testing SAP Leonardo Image Feature Extraction API (https://sandbox.api.sap.com/ml/featureextraction/inference_sync). I have the base64 string of the image and I want to transform it to a file object and zip it, then to send the zipped image file to this API using XMLHttpRequest. But the response text is "Service requires a list of (zipped) images".
I attach my HTTP request header and parameters in below screenshots.
Although we see a messy code in parameters, the zipped file Download here is created successfully.
If you cannot download the zipped file, please refer to the screenshot below.
Everything seems to be fine. However, the response text is as below with status 400.
My javascript code is shown below. What is wrong? It drives me crazy...
dataURItoBlob: function(dataURI, fileName) {
//convert base64/URLEncoded data component to raw binary data held in a string
var byteString;
if (dataURI.split(',')[0].indexOf('base64') >= 0)
byteString = atob(dataURI.split(',')[1]);
else
byteString = unescape(dataURI.split(',')[1]);
//separate out the mime component
var mimeString = dataURI.split(',')[0].split(':')[1].split(';')[0];
//write the bytes of the string to a typed array
var ia = new Uint8Array(byteString.length);
for (var i = 0; i < byteString.length; i++) {
ia[i] = byteString.charCodeAt(i);
}
var blob = new Blob([ia], {encoding:"UTF-8",type:mimeString});
//A Blob() is almost a File() - it's just missing the two properties below which we will add
blob.lastModifiedDate = new Date();
blob.name = fileName + '.' + mimeString.split('/')[1];
return blob;
},
onSubmit: function(oEvent) {
var oImage = this.getView().byId('myImage');
//oImage.getSrc() : 'data:image/png;base64,iVBORw0KGgo...'
var imageFile = this.dataURItoBlob(oImage.getSrc(), 'myImage');
var zip = new JSZip();
zip.file(imageFile.name, imageFile);
zip.generateAsync({
type:"blob",
compression: 'DEFLATE', // force a compression for this file
compressionOptions: {
level: 6,
},
}).then(function(content) {
//saveAs(content, "hello.zip");
// start the busy indicator
var oBusyIndicator = new sap.m.BusyDialog();
oBusyIndicator.open();
var formData = new FormData();
formData.append('files', content, 'myImage.zip');
var xhr = new XMLHttpRequest();
xhr.withCredentials = false;
xhr.addEventListener("readystatechange", function () {
if (this.readyState === this.DONE) {
oBusyIndicator.close();
//navigator.notification.alert(this.responseText);
console.log(this.responseText);
}
});
//setting request method
//API endpoint for API sandbox
//Destionation '/SANDBOX_API' in HCP is configured as 'https://sandbox.api.sap.com'
var api = "/SANDBOX_API/ml/featureextraction/inference_sync";
xhr.open("POST", api);
//adding request headers
xhr.setRequestHeader("Content-Type", "multipart/form-data");
xhr.setRequestHeader("Accept", "application/json");
//API Key for API Sandbox
xhr.setRequestHeader("APIKey", "yQd5Oy785NkAIob6g1eNwctBg4m1LGQS");
//sending request
xhr.send(formData);
});
},
I fix this issue by myself. I put my solution just for others' information. It's very easy and only below code needs to be removed before sending request. I have no idea why. Please suggest if you know the reason. Thanks in advance!
xhr.setRequestHeader("Content-Type", "multipart/form-data");
Best Regards,
Shelwin Wei
Basically that's it, Im trying to record audio, using the Web Audio API and tuna.js or pizzicato.js add some effects and save it into a server in base64.
The problem is, using recorder.js the audio is save the record in a blob object, after that I can transform it into a base64 (not editable) with the FileReader(). I've trying add effects to the blob:url, and works perfectly, but I cant transform the edited audio into base64.
recorder.addEventListener( "dataAvailable", function(e){
var fileName = new Date().toISOString() + "." + e.detail.type.split("/")[1];
var url = URL.createObjectURL( e.detail );
var audiob = "";
var reader = new window.FileReader();
reader.readAsDataURL(e.detail); //e.detail
reader.onloadend = function() {
base64data = reader.result;
audiob = base64data;
}
e.detail ->
Blob {size: 28278, type: "audio/wav"}
size:28278
type:"audio/wav"
__proto__:Blob
Any help? :)
PD: Sorry about my english, I know it sucks
I need to embed a Flash .swf on the page and am unable use the normal way of setting the src or data attribute to the swf url - don't ask :s. So, I'm doing an ajax request for the swf, converting to a blob and then generating a blob url which I set as the swf src. Then I realised that as I'm building with Grunt, there may be a way to just write the swf file into the code as a blob in a var, and avoid the ajax request completely. Here's the code with the ajax request:
function createFlashMovie(blobUrl){
var obj = document.createElement("object");
obj.setAttribute("width", "800");
obj.setAttribute("height", "600");
obj.setAttribute("type", "application/x-shockwave-flash");
obj.setAttribute("data", blobUrl);
document.body.appendChild(obj);
}
function onAjaxLoad(oResponse){
blobUrl = window.URL.createObjectURL(oResponse);
createFlashMovie(blobUrl);
};
//do the xhr request for a.swf
var xhr = new XMLHttpRequest();
xhr.onreadystatechange = function(){
if (this.readyState == 4 && this.status == 200){
onAjaxLoad(this.response);
}
}
xhr.open('GET', '//theserver.com/a.swf');
xhr.responseType = 'blob';
xhr.send();
...but I'm sure it must be possible to have something like this which is replaced by grunt to have the blob already available when it runs, and go straight to creating the blob url without the xhr request:
var theBlob = new Blob(["GRUNT_WRITES_THIS_IN_FROM_FILE"], {type: "application/x-shockwave-flash"});
Well, grunt is at its core just a Node program, so you can use any node command in your Gruntfile or tasks definitions. And it seems that Node's http.request would be perfect for your needs.
So:
add a custom task in your Gruntfile (http://gruntjs.com/creating-tasks#custom-tasks)
that uses http.request to download your swf (https://docs.nodejitsu.com/articles/HTTP/clients/how-to-create-a-HTTP-request)
update your code to use the local swf
I found a solution. Convert your swf file to be a base64-encoded string. At the moment I'm doing this separately and then pasting it into the source JS, but it can be automated at build time with grunt. Then in the page script create a var to store it as a dataURI:
var swfAsDataUri = "data:application/x-shockwave-flash;base64,BIG_LONG_CHUNK_OF_DATA_THAT_IS_YOUR_ENCODED_SWF_FILE__GRUNT_CAN_WRITE_THIS_IN_AT_BUILD_TIME";
Create a blob from the data url, and then create an object url from the blob:
//function taken from http://stackoverflow.com/questions/27159179/how-to-convert-blob-to-file-in-javascript
dataURLToBlob = function(dataURL) {
var BASE64_MARKER = ';base64,';
var parts = dataURL.split(BASE64_MARKER);
var contentType = parts[0].split(':')[1];
var raw = window.atob(parts[1]);
var rawLength = raw.length;
var uInt8Array = new Uint8Array(rawLength);
for (var i = 0; i < rawLength; ++i) {
uInt8Array[i] = raw.charCodeAt(i);
}
return new Blob([uInt8Array], {type: contentType});
};
var blobUrl = window.URL.createObjectURL( dataURLToBlob(swfAsDataUri) );
You can then use the object url as the src data for the flash movie's object tag when it's embedded:
function createFlashMovie(blobUrl){
var obj = document.createElement("object");
obj.setAttribute("width", "800");
obj.setAttribute("height", "600");
obj.setAttribute("type", "application/x-shockwave-flash");
obj.setAttribute("data", blobUrl); //use the object url here
document.body.appendChild(obj);
}
...and there you have it, no additional http request for the swf file.