Develop Reference

express.js: pass an uploaded image to s3

I am trying to pass an image uploaded from a react app through express to a managed s3 bucket. The platform/host I am using creates and manages the s3 bucket and generates upload and access urls. This all works fine (I have tested a generated upload url in postman with an image in a binary body and it worked perfectly).
My problem is passing the image through express. I am using multer to get the image from the form but I am assuming multer is turning that image into some kind of file object and s3 is expecting some sort of blob or stream.
In following code, the image in req.file exists, I get a 200 response from s3 with no errors and when I visit the asset url the url works, but the image itself is missing.
const router = Router();
const upload = multer()
router.post('/', upload.single('file'), async (req, res) => {
console.log(req.file)
const asset = req.file
const assetPath = req.headers['asset-path']
let s3URLs = await getPresignedURLS(assetPath)
const sendAsset = await fetch(
s3URLs.urls[0].upload_url, // the s3 upload url
{
method: 'PUT',
headers: {
"Content-Type": asset.mimetype
},
body: asset,
redirect: 'follow'
}
)
console.log("s3 response", sendAsset)
res.status(200).json({"url": s3URLs.urls[0].access_url });
});
export default router;
I am not sure what to do to convert what multer gives me to something that aws s3 will accept. I am also open to getting rid of multer if there is an easier way to upload binary files to express.

Instead of multer, you can use multiparty to get file data from request object. And to upload file to s3 bucket you can use aws-sdk.
const AWS = require("aws-sdk");
const multiparty = require("multiparty");
/**
* Helper method which takes the request object and returns a promise with a data.
*/
const getDataFromRequest = (req) =>
new Promise(async(resolve, reject) => {
const form = new multiparty.Form();
await form.parse(req, (err, fields, files) => {
if (err) reject(err);
const bucketname = fields.bucketname[0];
const subfoldername = fields.subfoldername[0];
const file = files["file"][0]; // get the file from the returned files object
if (!file) reject("File was not found in form data.");
else resolve({
file,
bucketname,
subfoldername
});
});
});
/**
* Helper method which takes the request object and returns a promise with the AWS S3 object details.
*/
const uploadFileToS3Bucket = (
file,
bucketname,
subfoldername,
options = {}
) => {
const s3 = new AWS.S3();
// turn the file into a buffer for uploading
const buffer = readFileSync(file.path);
var originalname = file.originalFilename;
var attach_split = originalname.split(".");
var name = attach_split[0];
// generate a new random file name
const fileName = name;
// the extension of your file
const extension = extname(file.path);
console.log(`${fileName}${extension}`);
const params = {
Bucket: bucketname, //Bucketname
ACL: "private", //Permission
Key: join(`${subfoldername}/`, `${fileName}${extension}`), // File name you want to save as in S3
Body: buffer, // Content of file
};
// return a promise
return new Promise((resolve, reject) => {
return s3.upload(params, (err, result) => {
if (err) reject(err);
else resolve(result); // return the values of the successful AWS S3 request
});
});
};
router.post('/', upload.single('file'), async(req, res) => {
try {
// extract the file from the request object
const {
file,
bucketname,
subfoldername
} = await getDataFromRequest(req);
// Upload File to specified bucket
const {
Location,
ETag,
Bucket,
Key
} = await uploadFileToS3Bucket(
file,
bucketname,
subfoldername
);
let response = {};
res["Location"] = Location;
response["ETag"] = ETag;
response["Bucket"] = Bucket;
response["Key"] = Key;
res.status(200).json(response);
} catch (error) {
throw error;
}
});
Request body will be form data with following fields
bucketname:
subfoldername:
file: FileData

For anyone that ever stumbles across this question the solution was to create an custom multer storage engine. Inside the engine you get access to the file with a stream property that s3 accepted (with the correct headers).

"Unsupported body payload object" when trying to upload to Amazon S3

I want to upload a file from my frontend to my Amazon S3 (AWS).
I'm using dropzone so I convert my file and send it to my backend.
In my backend my file is like:
{ fieldname: 'file',
originalname: 'test.torrent',
encoding: '7bit',
mimetype: 'application/octet-stream',
buffer: { type: 'Buffer', data: [Array] },
size: 7449 },
and when I try to upload my file with my function:
var file = data.patientfile.file.buffer;
var params = { Bucket: myBucket, Key: data.patientfile.file.fieldname, Body: file };
s3.upload(params, function (err, data) {
if (err) {
console.log("******************",err)
} else {
console.log("Successfully uploaded data to myBucket/myKey");
}
});
I get as error:
Unsupported body payload object
Do you know how can I send my file?
I have tried to send it with putobject and get a similar error.

I think you might need to convert the file content (which probably in this case is the data.patientfile.file.buffer) to binary
var base64data = new Buffer(data, 'binary');
so the params would be like:
var params = { Bucket: myBucket, Key: data.patientfile.file.fieldname, Body: base64data };
Or if I'm mistaken and the buffer is already in binary, then you can try:
var params = { Bucket: myBucket, Key: data.patientfile.file.fieldname, Body: data.patientfile.file.buffer};

This is my production code that is working.
Please note the issue can happen at data1111.
But, to get full idea, add all key parts of working code below.
client:
// html
<input
type="file"
onChange={this.onFileChange}
multiple
/>
// javascript
onFileChange = event => {
const files = event.target.files;
var file = files[0];
var reader = new FileReader();
reader.onloadend = function(e) {
// save this data1111 and send to server
let data1111 = e.target.result // reader.result // ----------------- data1111
};
reader.readAsBinaryString(file);
}
server:
// node.js/ javascript
const response = await s3
.upload({
Bucket: s3Bucket, // bucket
Key: s3Path, // folder/file
// receiving at the server - data1111 - via request body (or other)
Body: Buffer.from(req.body.data1111, "binary") // ----------------- data1111
})
.promise();
return response;
To make the above code working, it took full 2 days.
Hope this helps someone in future.

Implemented Glen k's answer with nodejs ...worked for me
const AWS = require('aws-sdk');
const s3 = new AWS.S3({
accessKeyId: process.env.AWSAccessKeyID,
secretAccessKey: process.env.AWSSecretAccessKey,
});
let base64data = Buffer.from(file.productImg.data, 'binary')
const params = {
Bucket: BUCKET_NAME,
Key: KEY,
Body: base64data
}
s3.upload(params, function(err, data) {
if (err) {
console.log(err)
throw err;
}
console.log(data)
console.log(`File uploaded successfully. ${data.Location}`);
})

Using signed requests with AWS S3 and uploading photos?

So I have a react native application that's kind of like slack and I'm trying to do image uploads to s3.
I went with getSignedUrl route.
So the client pics a photo,
fetches a signed url to the bucket
then changes the url on the server for that user
then a put request to the signed url that was fetched.
It mostly works the files get in the right bucket and they are photos. but
A) the link makes me download the file instead of displaying it in browser.
B) the file isn't an image...its an xml file and can only be opened in photoshop
I've tried changing the type in the data.append type,
Adding header to the signed request
Adding x-amz- headers to the signed request
hard coding the file type in server
converting image to base64 string with a native module but It still is coming up wrong.
Client Side calls to server
uploadToServer() {
// alert('coming soon!');
//Go back to profile page
this.props.navigation.goBack();
//grab user from navigator params
let user = this.props.navigation.state.params.user
let pic = this.state.selected;
// turn uri into base64
NativeModules.ReadImageData.readImage(pic.uri, (image) => {
console.log(image);
var data = new FormData();
data.append('picture', {
uri: image,
name: pic.filename,
type: 'image/jpeg'
});
//get the signed Url for uploading
axios.post(api.getPhotoUrl, {fileName: `${pic.filename}`}).then((res) => {
console.log("get Photo URL response", res);
//update the user with the new url
axios.patch(api.fetchUserByID(user.id), {profileUrl: res.data.url}).then((resp) => {
console.log("Update User response", resp.data);
}).catch(err => errorHandler(err));
//upload the photo using the signed request url given to me.
//DO I NEED TO TURN DATA INTO A BLOB?
fetch(res.data.signedRequest, {
method: 'PUT',
body: data
}).then((response) => {
console.log("UPLOAD PHOTO RESPONSE: ", response);
}).catch(err => errorHandler(err))
}).catch((err) => errorHandler(err))
})
}
GET SIGNED URL logic from on out
router.post('/users/sign-s3', (req, res) => {
const s3 = new aws.S3({signatureVersion: 'v4', region: 'us-east-2'});
const fileName = `${req.user.id}-${req.body.fileName}`;
const fileType = req.body.fileType;
const s3Params = {
Bucket: AWS_S3_BUCKET,
Key: `images/${fileName}`,
Expires: 60,
ContentType: 'image/jpeg',
ACL: 'public-read'
};
s3.getSignedUrl('putObject', s3Params, (err, data) => {
if (err) {
console.log(err);
return res.end();
}
const returnData = {
signedRequest: data,
url: `https://${AWS_S3_BUCKET}.s3.amazonaws.com/${s3Params.Key}`
};
res.write(JSON.stringify(returnData));
res.end();
return null;
});
});

You need to change content type from image to supported xml format if you want it to be displayed in browser.
Refer this and set content type accordingly.

Receiving "Uploading file too fast!" error from Imgur API

I created a node.js server that uses busboy to take requests, and pipe the files to Imgur for upload. However, I keep getting an "Uploading file too fast!" response from Imgur, and I'm not sure exactly what the problem is. Here is the code snippet involving busboy:
var express = require('express');
var Busboy = require('busboy');
var fs = require('fs');
var request = require('request-promise');
var router = express.Router();
router.post('/u', function(req, res, next) {
var busboy = new Busboy({headers: req.headers});
busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {
if(fieldname == 'image') {
var options = {
uri: 'https://api.imgur.com/3/image',
method: 'POST',
headers: {
'Authorization': 'Client-ID ' + clientID // put client id here
},
form: {
image: file,
type: 'file'
}
};
request(options)
.then(function(parsedBody) {
console.log(parsedBody);
})
.catch(function(err) {
console.log(err);
});
}
});
busboy.on('field', function(fieldname, val, fieldnameTruncated, valTruncated) {
console.log('field');
});
busboy.on('finish', function() {
res.status(200).end();
});
req.pipe(busboy);
});
As you can see I'm piping the request file directly into my request for imgur. Providing a ReadStream by simply saving the file to disc and then using fs.createReadStream() works perfectly, so I'm not really sure why trying to pipe directly from request to request gives me the error. The exact response I'm getting from Imgur is:
StatusCodeError: 400 - {"data":{"error":"Uploading file too fast!","request":"\/3\/image","method":"POST"},"success":false,"status":400}
If anyone has encountered this before, it would be helpful...

The first issue is that you should be using formData instead of form for file uploads. Otherwise, the request library won't send the correct HTTP request.
The second issue is that the stream object won't have the correct content length until it's fully processed. We can buffer the data ourselves and pass it after the initial file stream from busboy has processed.*
This gives us something that looks like
var express = require('express');
var Busboy = require('busboy');
var fs = require('fs');
var request = require('request-promise');
var router = express.Router();
router.post('/u', function(req, res, next) {
var busboy = new Busboy({headers: req.headers});
busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {
if(fieldname == 'image') {
// the buffer
file.fileRead = [];
file.on('data', function(data) {
// add to the buffer as data comes in
this.fileRead.push(data);
});
file.on('end', function() {
// create a new stream with our buffered data
var finalBuffer = Buffer.concat(this.fileRead);
var options = {
uri: 'https://api.imgur.com/3/image',
method: 'POST',
headers: {
'Authorization': 'Client-ID ' + clientID // put client id here
},
formData: {
image: finalBuffer,
type: 'file'
}
};
request(options)
.then(function(parsedBody) {
console.log(parsedBody);
})
.catch(function(err) {
console.log(err);
});
});
}
});
busboy.on('field', function(fieldname, val, fieldnameTruncated, valTruncated) {
console.log('field');
});
busboy.on('finish', function() {
res.status(200).end();
});
req.pipe(busboy);
});
Code for the buffering is from http://thau.me/2014/02/nodejs-streaming-files-to-amazons3/
Lastly, you may want to consider using the request library, as the request-promise library discourages the use of streams. See the github repo for more details: https://github.com/request/request-promise

save image in a folder using nodeJS and angularJS

save image in a folder using nodeJS and angularJS
Hi, i want save images in a folder, I have my backEnd in NodeJS and my frondEnd in AngularJS using ng-flow; my angular service code is the following
addImages : function (file) {
var uploadUrl = localPath + "addImages";
var fd = new FormData();
fd.append('file', file);
return $http.post(uploadUrl, fd, {
transformRequest: angular.identity,
headers: { 'Content-Type': undefined,
'file-name' : undefined,
'file-location' : undefined,
'user-id' : undefined
}
});
}
and i have the following POST service in Node
router.post('/addImages', function (req, res) {
console.log(req);
var tempPath = req.files.file.path;
console.log(tempPath);
var targetPath = path.resolve('/../static/image/image.png');
console.log(targetPath);
console.log(req.files.file.name);
if (path.extname(req.files.file.name).toLowerCase() === '.png') {
fs.rename(tempPath, targetPath, function(err) {
if (err) throw err;
console.log("Upload completed!");
});
} else {
fs.unlink(tempPath, function () {
if (err) throw err;
console.error("Only .png files are allowed!");
});
}
// ...
});
the error that i get is POST /assetImage/addImages 500
someone know a more easy form to get a image in angular and save in a folder using node?
Edit
when I print file using console.log i get o file object like File { name: "howls-moving-castle-40994-1920x1200…", lastModified: 1431122408000, lastModifiedDate: Date 2015-05-08T22:00:08.000Z, size: 910465, type: "image/jpeg" } but when print the fd get a empty object FormData { } I intentend pass file whitout using formData but in node get a Empty object to

If you are using Express.js, body-parser does not handle multipart bodies, due to their complex and typically large nature. For multipart bodies, you may be interested in the following modules:
busboy
multiparty
formidable
multer
See the suggested modules here for parsing multipart bodies.