I've built a chat server using rails, socket.io, and node that allows users to send images to each other, which I'm trying to get to work with Amazon s3. Here are the relevant code snippets from the client and server:
Client:
var img_file = this.files[0]; //retrieved from file input
file_reader.onload = function(e) {
//console.log(img_file);
var extension = img_file.name.split('.');
extension = extension[extension.length-1];
get_user_id(username, function(to_id) {
socket.emit('message', {
from: "<%= #user.displayname %>",
fromid: <%= #user.id %>,
to: username,
toid: parseInt(to_id),
content: e.target.result,
type: 'image',
extension: extension,
size: img_file.size
});
})
};
file_reader.readAsBinaryString(img_file);
Server:
socket.on('message', function(message) {
console.log(message.type + " message received, sending to appropriate person: " + message.to);
if (message.type == 'text') {
//send message like normal
sendMessage(message);
}
else {
//it's an image, need additional processing
var blank_file = uuid.v4() + '-' + new Date().getTime() + '.' + message.extension;
var contents = message.content;
s3.putObject({
Bucket: 'bucket-name', //not the real bucket name
Key: blank_file,
ContentType: 'image/' + message.extension,
//ContentLength: message.size,
Body: contents
},
function(err, data) {
if (err) {
console.log(err)
}
else {
console.log('file uploaded to s3 successfully');
message.content = getS3Url(blank_file);
sendMessage(message);
}
})
}
}
Whenever I try to upload an image using this code, it appears to work, but when I check my s3 bucket and open up the image, my operating system says that the file is unreadable or corrupt. I've tried using the ContentLength parameter but when I use that I get the error "BadDigest: The Content-MD5 you specified did not match what we received."
Can anyone help me out? I got this code to work properly without s3, but for some reason I keep getting corrupt or unrecognizable files since I started using it.
I figured out what was wrong. You need to use a binary buffer to pass the data to s3 instead of just passing the string as is, so basically use these for the s3 parameters:
s3.putObject({
Bucket: 'bucket-name',
Key: file,
Body: new Buffer(contents, 'binary') //use a buffer to pass the data over
}, function() { });
By using the buffer, it should upload the files properly.
Related
I was successful in storing images in mongodb using the base64 url. But when I tried to do so with PDFs, it gave an url which does not work. Wait, let me explain, when I put the image base64 url in the req.body of the POST request, the special signs would get disappeared, so I tried encodeURIComponent() method to make it error free. After that I found that storing the huge string in mongodb was too short to fit in the db, so I tried: app.use(express.json({limit: '50mb'})); app.use(express.urlencoded({limit: '50mb', extended: false })); and It worked! but when the client requests the base64 url, it would come encoded, so I put decodeURIComponent() to decode it and was not a great issue nd I got the desired result, yet with the Image one.
The main issue issue is when it comes to PDF. I don't know why it's happening with PDF only! when I make base64 url in CLIENT side and test it, it works fine, but when it comes to server side, all the mess happens. please help me deal with this.
Note: "I don't want to use Gridfs, formidabe, multer etc for file things"
here's my piece of code:
$('#seasonForm').submit(async function (e) {
e.preventDefault();
const form = $(this);
const ImgFile = document.getElementById('seasonThumbnail').files[0];
const PDFFile = document.getElementById('seasonPDF').files[0];
const imgurl = encodeURIComponent(await getBase64(ImgFile));
const PDFurl = encodeURIComponent(await getBase64(PDFFile));
const url = '/uploadSeason';
$.ajax({
type: "POST",
url: url,
data: form.serialize()+`&Version=<%- NxtSeasons %>&image=${imgurl}&PDF=${PDFurl}`,
success: data => {
console.log(data.message);
if (data.status == "error") {
showIt(".alert", data.message, "error");
} else {
showIt(".alert", data.message, "success");
}
}
});
})
wait, now don't get confused with getBase64() and showIt. these are my functions. getBase64() is a promice which returns base64 url of the file and showIt() is type of alert which I made. Now if you don't know what is base64 url, this is the getBase64 one's code:
const getBase64 = (file) => {
return new Promise((resolve, reject) => {
const reader = new FileReader();
reader.readAsDataURL(file);
reader.onload = () => resolve(reader.result);
reader.onerror = error => reject(error);
});
}
My nodejs code:
app.post("/uploadSeason", admin, async (req, res) => {
try {
const { Name, Desctiption, Version, image, PDF } = req.body;
const newSeason = new Season({
Name,
Desctiption,
Version,
image: encodeURIComponent(image),
PDF: encodeURIComponent(PDF),
});
await newSeason.save();
res.json({
status: "success",
message: "Season added successfully"
});
} catch (e) {
console.log(e);
res.json({
status: "error",
message: e
});
}
});
I am trying to upload buffer data from an image into S3. It gets uploaded fine. But when I try to download/view the image in S3, it throws an error. I have tried the following -
The image is available to me in Buffer format (JSON). I cannot change this
let image = { "type": "Buffer", "data": [45, 45, 45....]
let buf = new Buffer(image )
let params = {
Bucket: "bucketName",
Key: "TestImage123haha.PNG",
Body: buf ,
ACL: 'public-read',
ContentType: 'image/jpeg'
};
s3.upload(params, function(err, data) {
if (err) {
console.log('ERROR MSG: ', err);
} else {
console.log('Successfully uploaded data' + data.Location);
}
})
After upoading the image if I try to visit the URL of the bucket where it is stored and view the image, this is what I get -
When I print the buffer data - buf in console , this is what I get -
This problem took me many days but I was able to solve it:
In your API GATE WAY of funtion lambda go to Configuration > Binary types.
Add multipart/form-data
In my electron app I have a button that says "upload contract".
When clicked it looks from the stored contract's filename ie. /home/users/filename.docx and gets it (no need for a dialog window).
Now, I have problems uploading this file as a multipart form data while using axios.
I've read about this issue, asking for file upload on axios which led to this pull request for file upload on browser and this pull for uploading in node.js.
I've read through the issues and comments applying bit by bit but seems to have troubles getting the real file uploading right using axios.
Here is what I have been doing.
Upload as browser
export function generateContract(fileUrl, customer) {
const data = new FormData()
const file = fs.createReadStream(fileUrl)
data.append('names', customer.names)
data.append('email', customer.email)
data.append('contract', file)
return data
}
//- In between generateContract & uploadFile function some function
//- gets the generatedContract data & push it to uploadFile for axios uploading.
//- In API file.
export function uploadFile(formData) {
return checkConnetion()
.then(() => {
return axios.post(emailUrl, formData)
})
.catch(() => {
return axios.post(emailUrlOffline, formData)
})
}
In generatedContract I'm using FormData that (I assume) is a global function in browsers to create a form instance for uploading files.
I'm also using fs.createReadStream cause without it my file upload looks like an object with three strings (the fileUrl gets to be a normal string).
However, uploading like this and console.loging my request body from my local server I get a hopeless
{ names: 'Mwajuma Salmin',
email: 'sammwaj#yahoo.com',
contract: '[object Object]' }
The contract file is `[object Object]'.
Uploading as node.js
Everything is the same, except I'm using npm's form-data.
Output is.
{ '[object FormData]': '' }
Where am I messing up?
I tried uploading with postman and manually selected the file to upload as form-data and that other data (email & names), it was all successful!
Output.
{ names: 'maotora',
contract:
[ File {
domain: null,
_events: {},
_eventsCount: 0,
_maxListeners: undefined,
size: 11609,
path: '/tmp/upload_0af5c038e147888f0a7b89ad4784a4dc',
name: 'Don Joe_08-06-17\'s_contract-for Plot 011.pdf',
type: 'application/pdf',
hash: null,
lastModifiedDate: 2017-09-28T18:56:31.083Z,
_writeStream: [Object] } ] }
I can't use form uploading in the app (just the button to upload without opening a dialog to select file) I only have the filename to the file.
Thanks.
This question has been open for a sometime since I couldn't solve the issue with axios and had to use superagent for a while.
However, now that it's solved and can use axios just fine, this is how I did solved it on my project.
//- Correctly placing a file_Url into FormData
export function generateContract(url, {names, email}) {
const file = {
uri: url,
name: _.replace(url, /\.[doc,docx,odt,]+$/, ''),
type: 'docx'
}
const formData = new FormData()
formData.append('contract', file)
formData.append('names', names)
formData.append('email', email)
return formData
}
//- 1. Calling generateContract method
//- 2. Passing the formdata (from method 1) to sendEmail
const contractData = generateContract(contractUrl, payload)
const response = yield sendEmail(contractData)
//- The sendEmail function that makes requests using axios
export function sendEmail(contractData) {
return checkConnetion()
.then(() => {
return axios.post(emailUrl, contractData)
})
.catch(() => {
// return axios.post(emailUrlOffline, contractData)
userLog('Cannot connect to server, check your internet settings.', 'Connection Error', 'error')
})
}
I'm doing an application with react-native. Now I'm trying to send an image from the mobile to the server (Node Js). For this I'm using react-native-image-picker. And the problem is that when I send the image it save a file but it's empty not contain the photo. I think that the problem probably is that the server can't access to the path of the image because is in a different device. But I don't know how I can do it.
React-Native:
openImagePicker(){
const options = {
title: 'Select Avatar',
storageOptions: {
skipBackup: true,
path: 'images'
}
}
ImagePicker.showImagePicker(options, (imagen) =>{
if (imagen.didCancel) {
console.log('User cancelled image picker');
}
else if (imagen.error) {
console.log('ImagePicker Error: ', imagen.error);
}
else if (imagen.customButton) {
console.log('User tapped custom button: ', imagen.customButton);
}
else {
let formdata = new FormData();
formdata.append("file[name]", imagen.fileName);
formdata.append("file[path]", imagen.path);
formdata.append("file[type]", imagen.type);
fetch('http://X/user/photo/58e137dd5d45090d0b000006', {
method: 'PUT',
headers: {
'Content-Type': 'multipart/form-data'
},
body: formdata
})
.then(response => {
console.log("ok");
})
.catch(function(err) {
console.log(err);
})
}})}
Node Js:
addPhotoUser = function (req, res) {
User.findById(req.params.id, function(err, user) {
fs.readFile(req.body.file.path, function (err, data) {
var pwd = 'home/ubuntu/.../';
var newPath = pwd + req.body.file.name;
fs.writeFile(newPath, data, function (err) {
imageUrl: URL + req.body.file.name;
user.save(function(err) {
if(!err) {
console.log('Updated');
} else {
console.log('ERROR: ' + err);
}
res.send(user);
});
});
});
});
};
Yes, the problem is that the filepath is on the local device and not the server. You want to send the actual data returned to you by react-native-image-picker not the uri. It looks like that library encodes the data with base64 so you're going to want send that to your server, not the uri returned from the library because it won't be accessible on a remote server.
What this means is that you won't be reading any files on your server but instead just decoding a base64 string in the response body and writing that to your filesystem.
For the client side:
let formdata = new FormData();
formdata.append("file[name]", imagen.fileName);
formdata.append("file[data]", imagen.data); // this is base64 encoded!
formdata.append("file[type]", imagen.type);
fetch('http://X/user/photo/58e137dd5d45090d0b000006', {
method: 'PUT',
headers: {
'Content-Type': 'multipart/form-data'
},
body: formdata
})
On the server side atob to decode from base64 before writing to the filesystem:
let decoded = atob(req.body.data)
// now this is binary and can written to the filesystem
From there:
fs.writeFile(newPath, decoded, function (err) {
imageUrl: newPath;
user.save(function(err) {
if(!err) {
console.log('Updated');
} else {
console.log('ERROR: ' + err);
}
res.send(user);
});
});
Note, you don't need the filesystem write that's in your code because you're decoding the image that was sent as a b64 string in your request.
There also seems to be some oddities with how you're using that user object. You seem to be only passing a function that handles errors and not any actual data. I don't know what ORM you're using so it's hard to say how it should work. Maybe something like this?
user.save({imageUrl:uriReturnedByFsWrite}, (err, data)=>{...})
Good luck :)
Make an object then send that object to the server. The object will consist of name,path and type, like this:
var imageData = {name: 'image1', path: uri, type: 'image/jpeg'}
Above is a one way to send the image data. The other way is to convert it into BLOB so that server side programmer doesn't have to do this task on their end. You can make BLOB by use of react-native-fetch-blob.
One more way is to directly upload the images to the amazon server(s3) and send the link to the backend..
Function that returns base64 string:
var RNFetchBlob = require('react-native-fetch-blob').default;
getImageAttachment: function(uri_attachment, mimetype_attachment) {
return new Promise((RESOLVE, REJECT) => {
// Fetch attachment
RNFetchBlob.fetch('GET', config.apiRoot+'/app/'+uri_attachment)
.then((response) => {
let base64Str = response.data;
var imageBase64 = 'data:'+mimetype_attachment+';base64,'+base64Str;
// Return base64 image
RESOLVE(imageBase64)
})
}).catch((error) => {
// error handling
console.log("Error: ", error)
});
},
Cheers :)
I'm integrating Quickblox in my Ionic2 app, for now I was able to do all the things except uploading a file.
In Quickblox you have to upload a file using a function made by them that according to js documentation looks like this:
var inputFile = $("input[type=file]")[0].files[0];
var params = {name: inputFile.name,
file: inputFile,
type: inputFile.type,
size: inputFile.size,
public: false};
QB.content.createAndUpload(params, function(err, response){
if (err) {
console.log(err);
} else {
console.log(response);
var uploadedFile = response;
var uploadedFileId = response.id;
}
});
So I translated above code to typescript and I have something like this:
uploadFile(filename) {
File.resolveDirectoryUrl(cordova.file.dataDirectory).then(
(dirEntry) => {
File.getFile(dirEntry, filename, { create: false }).then(
(fileEntry) => {
console.log(fileEntry);
fileEntry.file((file) => {
console.log(file);
var params = {
name: file['name'],
file: file,
type: file['type'],
size: file['size'],
'public': false
};
quickblox.content.createAndUpload(params,
(err, response) => {
if (err) {
console.log(err);
} else {
console.log(response);
var uploadedFileId = response.id;
var msg = {
type: 'groupchat',
body: "[attachment]",
extension: {
save_to_history: 1,
}
};
msg["extension"]["attachments"] = [{ id: uploadedFileId, type: 'photo' }];
quickblox.chat.send(this.xmpp_room_jid, msg);
}
});
})
}).catch(
(err) => {
console.log(err);
}
);
}
);
}
This work in the terms of "I get ok responses from quickblox server", but when I go to the admin console of quickblox to check the uploaded content I find out that the image I uploaded has 0 bytes.
So after checking the code for a while I compared side by side all my function calls with the example app of quickblox and the only difference I could find was in the File object.
This is the File object I get in my Ionic 2 app:
And this is the one I get in the quickblox js example:
All the others things looks identically except this File object.
I'm almost sure that this is the problem I'm having, and because I'm very newbie in this field, I couldn't find a way to cast from my File object in Ionic to something like the File object in the js example.
Thanks in advance at all for your time and help.
EDIT:
I attach the requests/responses logs from my Ionic app:
Could you please post the code you used to connect to chat, create a session, open a video call?
The documentation on quickblox is very bad and i got stuck at connecting to chat.