I am using this code I found on the internet to upload multiple files to an Amazon S3 server.
const AWS = require("aws-sdk"); // from AWS SDK
const fs = require("fs"); // from node.js
const path = require("path"); // from node.js
// configuration
const config = {
s3BucketName: 'your.s3.bucket.name',
folderPath: '../dist' // path relative script's location
};
// initialize S3 client
const s3 = new AWS.S3({ signatureVersion: 'v4' });
// resolve full folder path
const distFolderPath = path.join(__dirname, config.folderPath);
// get of list of files from 'dist' directory
fs.readdir(distFolderPath, (err, files) => {
if(!files || files.length === 0) {
console.log(`provided folder '${distFolderPath}' is empty or does not exist.`);
console.log('Make sure your project was compiled!');
return;
}
// for each file in the directory
for (const fileName of files) {
// get the full path of the file
const filePath = path.join(distFolderPath, fileName);
// ignore if directory
if (fs.lstatSync(filePath).isDirectory()) {
continue;
}
// read file contents
fs.readFile(filePath, (error, fileContent) => {
// if unable to read file contents, throw exception
if (error) { throw error; }
// upload file to S3
s3.putObject({
Bucket: config.s3BucketName,
Key: fileName,
Body: fileContent
}, (res) => {
console.log(`Successfully uploaded '${fileName}'!`);
});
});
}
});
How can I get notified that the upload is done to execute another process? res is called when a single file were successfully uploaded.
How about incrementing a counter when a file uploads and then checking if all files have been uploaded:
...
var uploadCount = 0
// Read file contents
fs.readFile(filePath, (error, fileContent) => {
// If unable to read file contents, throw exception
if (error) { throw error }
// Upload file to S3
s3.putObject({
Bucket: config.s3BucketName,
Key: fileName,
Body: fileContent
}, (res) => {
console.log(`Successfully uploaded '${fileName}'!`)
// Increment counter
uploadCount++
// Check if all files have uploaded
// 'files' provided in callback from 'fs.readdir()' further up in your code
if (uploadCount >= files.length) {
console.log('All files uploaded')
}
})
})
...
You could try using promises and promise.all
const AWS = require("aws-sdk"); // from AWS SDK
const fs = require("fs"); // from node.js
const path = require("path"); // from node.js
// configuration
const config = {
s3BucketName: 'your.s3.bucket.name',
folderPath: '../dist' // path relative script's location
};
// initialize S3 client
const s3 = new AWS.S3({ signatureVersion: 'v4' });
// resolve full folder path
const distFolderPath = path.join(__dirname, config.folderPath);
// get of list of files from 'dist' directory
fs.readdir(distFolderPath, (err, pathURLS) => {
if(!pathURLS || pathURLS.length === 0) {
console.log(`provided folder '${distFolderPath}' is empty or does not exist.`);
console.log('Make sure your project was compiled!');
return;
}
let fileUploadPromises = pathURLS.reduce(uplaodOnlyFiles, []);
//fileUploadPromises.length should equal the files uploaded
Promise.all(fileUploadPromises)
.then(() => {
console.log('All pass');
})
.catch((err) => {
console.error('uploa Failed', err);
});
});
function uploadFileToAWS(filePath) {
return new Promise(function (resolve, reject) {
try {
fs.readFile(filePath, function (err, buffer) {
if (err) reject(err);
// upload file to S3
s3.putObject({
Bucket: config.s3BucketName,
Key: filePath,
Body: buffer
}, (res) => {
resolve(res)
});
});
} catch (err) {
reject(err);
}
});
}
function uplaodOnlyFiles(fileUploadPromises, pathURL) {
const fullPathURL = path.join(distFolderPath, pathURL);
if (!fs.lstatSync(fullPathURL).isDirectory()) {
fileUploadPromises.push(uploadFileToAWS(fullPathURL));
}
return fileUploadPromises;
}
Related
DETAILS:
I have an express route which is set up to accept files as multipart/formdata and upload them to an S3 bucket. I am using multer to filter image types, as well as store them temporarily on the server through the creation of an upload directory. The files are removed shortly after upload success. The array of files are named images as per multer configuration, and accepts a maximum of 3 images.
The code works perfectly on my local machine. I test through POSTMAN and can upload 1-3 files and get the proper response. If there are no files attached, the correct response is triggered as well, all with status code 200.
PROBLEM:
The exact same codebase is deployed on Amazon ECS with Docker, but somehow keeps failing consistently with status code 500 and a generic 'Error' message that is not found in the codebase. Using logs I have determined that multer is not the cause, as it passes through the filter. It appears to be failing somewhere between the multer middleware and the route itself, with an exception.
Exception: Using POSTMAN, if a multipart/formdata POST request is made with no files I.E empty images array, the route is triggered properly and the message "You did not attach any images" is returned as a response.
I have been unable to figure out the issue and appreciate it if some guidance can be provided on this issue!
CODE SNIPPETS:
filesController:
files.post(
"/multiple",
upload.array("images", 3),
async (req: ExpressRequest, res: ExpressResponse) => {
try {
const files: { [fieldname: string]: Express.Multer.File[] } | Express.Multer.File[] =
req.files;
console.log("FILES", files);
// execute only if there are files
if (files.length > 0) {
const dataPromises = (files as Array<Express.Multer.File>).map(
async (file: Express.Multer.File) => {
// check if file.mimetype here is 'image/heic', and convert into jpeg accordingly
const fileNameWithoutExt = file.filename.split(".")[0];
try {
if (file.mimetype == "image/heic") {
await convertFile(file, fileNameWithoutExt, 0.2);
const response = await uploadFilePath(
S3_IMAGE_BUCKET,
`./uploads/${fileNameWithoutExt}.jpeg`,
`${fileNameWithoutExt}.jpeg`
);
console.log("HEIC File Upload Response", response);
fs.unlinkSync(`./uploads/${fileNameWithoutExt}.jpeg`);
fs.unlinkSync(file.path);
return {
fileName: `${fileNameWithoutExt}.jpeg`,
metaData: response.$metadata,
};
} else {
const response = await uploadFile(S3_IMAGE_BUCKET, file);
console.log("JPEG File Upload Response", response);
fs.unlinkSync(file.path);
return {
fileName: file.filename,
metaData: response.$metadata,
};
}
} catch (err) {
console.error("Error for file conversion/upload", err, err.stack);
res.status(500).send({
message: "Upload failed due to conversion or something.",
error: err,
stack: err.stack,
});
}
}
);
const fileData = await Promise.all(dataPromises);
const fileNames = fileData.map((data: any) => data.fileName);
const statusCodes = fileData.map((data: any) => data.metaData.httpStatusCode);
if (statusCodes.find((statusCode) => statusCode === 200)) {
res.status(200).send({
filePath: `/image/`,
fileNames,
});
} else {
res.status(403).send({
message: "Upload failed. Please check credentials or file has been selected.",
});
}
} else {
res.status(200).send({
message: "You did not attach any images",
});
}
} catch (err) {
res.status(500).send({
message: "Upload failed. Please check credentials or file has been selected.",
});
}
}
);
multer configuration:
const storage = multer.diskStorage({
// potential error, path to store files, callback
destination: (req, file, cb) => {
// cb acceptes two arguments: 1. err 2. destination folder wrt to server.js
cb(null, "uploads/");
},
filename: (req, file, cb) => {
console.log("MULTER STORAGE STARTED")
const date = new Date().toISOString().substring(0, 10);
// const name = `${req.body.first_name}_${req.body.last_name}`;
// cb defines the name of the file when stored
const alphabet = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz-";
const nanoid = customAlphabet(alphabet, 20);
cb(null, `${date}_${nanoid()}_${file.originalname}`);
console.log("FILE NAME CREATED, MULTER STORAGE STOPPED")
},
});
/* Accept jpeg or png files only */
// NOTE: file type rejection works, but there is no error message displayed if file is rejected. logic in route continues to be executed
const fileFilter = (
req: Request,
file: Express.Multer.File,
cb: (error: Error | null, accepted: boolean) => void
) => {
console.log("======== FILE FILTER ========", file);
if (
file.mimetype === "image/jpeg" ||
file.mimetype === "image/png" ||
file.mimetype === "image/heic"
) {
cb(null, true);
console.log("FILTER PASSED")
} else {
console.log("FILTER FAILED");
cb(null, false);
}
};
/* Only accepts filesize up to 5MB */
// the first parameter is super important that determines where the data is stored on the server
const upload = multer({
dest: "uploads/", // default simple config to upload file as binary data
storage, // enable if storage of actual file is required.
// limits: { fileSize: 1024 * 1024 * 5 },
fileFilter,
});
SCREENSHOTS:
response with no images in form data
response with images in form data
Can you make sure the upload directory exists in your Docker container? Multer will not create it if it doesn't exist. It might be failing silently between your storage function and the actual writing of the files to disk.
const storage = multer.diskStorage({
// potential error, path to store files, callback
destination: (req, file, cb) => {
// cb acceptes two arguments: 1. err 2. destination folder wrt to server.js
cb(null, "uploads/");
},
should be something like:
import { access, constants } from "fs";
import { join } from "path";
...
const storage = multer.diskStorage({
// potential error, path to store files, callback
destination: (req, file, cb) => {
// cb acceptes two arguments: 1. err 2. destination folder wrt to server.js
const currentDirectory: string = process.cwd();
const uploadDirectory: string = join(currentDirectory, 'uploads/');
// can we write to this path?
access(uploadDirectory, constants.W_OK, (err) => {
if (err) {
console.error(err);
cb(err, null);
}
cb(null, uploadDirectory);
})
},
I am trying to pass an image uploaded from a react app through express to a managed s3 bucket. The platform/host I am using creates and manages the s3 bucket and generates upload and access urls. This all works fine (I have tested a generated upload url in postman with an image in a binary body and it worked perfectly).
My problem is passing the image through express. I am using multer to get the image from the form but I am assuming multer is turning that image into some kind of file object and s3 is expecting some sort of blob or stream.
In following code, the image in req.file exists, I get a 200 response from s3 with no errors and when I visit the asset url the url works, but the image itself is missing.
const router = Router();
const upload = multer()
router.post('/', upload.single('file'), async (req, res) => {
console.log(req.file)
const asset = req.file
const assetPath = req.headers['asset-path']
let s3URLs = await getPresignedURLS(assetPath)
const sendAsset = await fetch(
s3URLs.urls[0].upload_url, // the s3 upload url
{
method: 'PUT',
headers: {
"Content-Type": asset.mimetype
},
body: asset,
redirect: 'follow'
}
)
console.log("s3 response", sendAsset)
res.status(200).json({"url": s3URLs.urls[0].access_url });
});
export default router;
I am not sure what to do to convert what multer gives me to something that aws s3 will accept. I am also open to getting rid of multer if there is an easier way to upload binary files to express.
Instead of multer, you can use multiparty to get file data from request object. And to upload file to s3 bucket you can use aws-sdk.
const AWS = require("aws-sdk");
const multiparty = require("multiparty");
/**
* Helper method which takes the request object and returns a promise with a data.
*/
const getDataFromRequest = (req) =>
new Promise(async(resolve, reject) => {
const form = new multiparty.Form();
await form.parse(req, (err, fields, files) => {
if (err) reject(err);
const bucketname = fields.bucketname[0];
const subfoldername = fields.subfoldername[0];
const file = files["file"][0]; // get the file from the returned files object
if (!file) reject("File was not found in form data.");
else resolve({
file,
bucketname,
subfoldername
});
});
});
/**
* Helper method which takes the request object and returns a promise with the AWS S3 object details.
*/
const uploadFileToS3Bucket = (
file,
bucketname,
subfoldername,
options = {}
) => {
const s3 = new AWS.S3();
// turn the file into a buffer for uploading
const buffer = readFileSync(file.path);
var originalname = file.originalFilename;
var attach_split = originalname.split(".");
var name = attach_split[0];
// generate a new random file name
const fileName = name;
// the extension of your file
const extension = extname(file.path);
console.log(`${fileName}${extension}`);
const params = {
Bucket: bucketname, //Bucketname
ACL: "private", //Permission
Key: join(`${subfoldername}/`, `${fileName}${extension}`), // File name you want to save as in S3
Body: buffer, // Content of file
};
// return a promise
return new Promise((resolve, reject) => {
return s3.upload(params, (err, result) => {
if (err) reject(err);
else resolve(result); // return the values of the successful AWS S3 request
});
});
};
router.post('/', upload.single('file'), async(req, res) => {
try {
// extract the file from the request object
const {
file,
bucketname,
subfoldername
} = await getDataFromRequest(req);
// Upload File to specified bucket
const {
Location,
ETag,
Bucket,
Key
} = await uploadFileToS3Bucket(
file,
bucketname,
subfoldername
);
let response = {};
res["Location"] = Location;
response["ETag"] = ETag;
response["Bucket"] = Bucket;
response["Key"] = Key;
res.status(200).json(response);
} catch (error) {
throw error;
}
});
Request body will be form data with following fields
bucketname:
subfoldername:
file: FileData
For anyone that ever stumbles across this question the solution was to create an custom multer storage engine. Inside the engine you get access to the file with a stream property that s3 accepted (with the correct headers).
I'm trying to upload simple documents using busboy / express.js to google cloud.
I am getting this error.
Error: Cannot find module 'busboy/lib/types/node_modules/dicer'
and this is the code for the request.
// Upload a document for claim
exports.uploadDocument = (req, res) => {
const BusBoy = require("busboy");
const path = require("path");
const os = require("os");
const fs = require("fs");
const busboy = new BusBoy({ headers: req.headers });
let DocumentToBeUploaded = {};
let DocumentFileName;
// change this section to storing pdfs and docs etc
busboy.on("file", (fieldname, file, filename) => {
console.log(fieldname, file, filename);
const documentExtension = filename.split(".")[
filename.split(".").length - 1
];
// 32756238461724837.png
DocumentFileName = `${Math.round(
Math.random() * 1000000000000
).toString()}.${documentExtension}`;
const filepath = path.join(os.tmpdir(), DocumentFileName);
DocumentToBeUploaded = { filepath, mimetype };
file.pipe(fs.createWriteStream(filepath));
});
busboy.on("finish", () => {
admin
.storage()
.bucket()
.upload(DocumentToBeUploaded.filepath, {
resumable: false,
metadata: {
metadata: {
contentType: DocumentToBeUploaded
}
}
})
.then(() => {
const docUrl = `https://firebasestorage.googleapis.com/v0/b/${config.storageBucket}/o/${DocumentFileName}?alt=media`;
return db.doc(`/users/${req.Claim.ClaimId}`).update({ docUrl });
})
.then(() => {
return res.json({ message: "document uploaded successfully" });
})
.catch(err => {
console.error(err);
return res.status(500).json({ error: "something went wrong" });
});
});
busboy.end(req.rawBody);
};
Just trying to upload a very simple text document currently. Surely it can't be that difficult and I'm making a simple mistake somewhere.
Appreciate the help :)
You need to install busyboy:
npm i busboy
You can find more about this npm package in the following link:
https://www.npmjs.com/package/busboy
I am working with the GridFS library in express and node. I'm trying to create multiple buckets. For example I already have a bucket titled avatars, which stores images.
/* Start of mongo connection for uploading files */
const mongoURI = "mongodb://localhost:27017/PTAdata";
const conn = mongoose.createConnection(mongoURI);
let gfs;
conn.once('open', () => {
gfs = stream(conn.db, mongoose.mongo);
gfs.collection('avatars');
})
const storage = new GridFs({
url: "mongodb://localhost:27017/PTAdata",
file: (req, file) => {
return new Promise((resolve, reject) => {
crypto.randomBytes(16, (err, buf) => {
if (err) {
return reject(err);
}
file.user = req.body.username
const name = file.originalname
const filename = buf.toString('hex') + path.extname(file.originalname);
const fileInfo = {
filename: file.user,
bucketName: 'avatars'
};
resolve(fileInfo);
});
});
}
});
const upload = multer({ storage });
I now want to create another bucket called audio which will store mp3 files. I checked the documentation for GridFS at https://docs.mongodb.com/manual/core/gridfs/ and it states that "you can choose a different bucket name, as well as create multiple buckets in a single database." However it does not provide any insight or steps to how. Has anyone done any work with the GridFS library and know how to create multiple buckets?
You need to store another "new GridFS" object in a different variable, than pass it to multer as a different storage property. In your case, this should work:
const storage = new GridFs({
url: "mongodb://localhost:27017/PTAdata",
file: (req, file) => {
return new Promise((resolve, reject) => {
crypto.randomBytes(16, (err, buf) => {
if (err) {
return reject(err);
}
file.user = req.body.username
const name = file.originalname
const filename = buf.toString('hex') + path.extname(file.originalname);
const fileInfo = {
filename: file.user,
bucketName: 'avatars'
};
resolve(fileInfo);
});
});
}
});
const anotherStorage = new GridFs({
url: "mongodb://localhost:27017/PTAdata",
file: (req, file) => {
return new Promise((resolve, reject) => {
crypto.randomBytes(16, (err, buf) => {
if (err) {
return reject(err);
}
file.user = req.body.username
const name = file.originalname
const filename = buf.toString('hex') + path.extname(file.originalname);
const fileInfo = {
filename: file.user,
bucketName: 'mp3files'
};
resolve(fileInfo);
});
});
}
});
const upload = multer({ storage });
const uploadSongs = multer({ storage: anotherStorage });
Finally, you should choose between those buckets accordingly to your endpoints, for example:
app.post('/api/uploadAvatar', upload.any(), (req, res)=> {
... do stuff
}
app.post('/api/uploadMp3', uploadSongs.any(), (req, res)=> {
... do stuff
}
For me, it made sense to change gfs.collection() in each case (inside the file: (req, file) function), but it worked without changing as well. Be aware that any() is just an option, an it's not the safest one, but it's great for testing your code.
I'm attempting to handle file uploads using a Google Cloud Function. This function uses Busboy to parse the multipart form data and then upload to Google Cloud Storage.
I keep receiving a ERROR: { Error: ENOENT: no such file or directory, open '/tmp/xxx.png' error when triggering the function.
The error seems to occur within the finish callback function when storage.bucket.upload(file) attempts to open the file path /tmp/xxx.png.
Example code
const path = require('path');
const os = require('os');
const fs = require('fs');
const Busboy = require('busboy');
const Storage = require('#google-cloud/storage');
const moment = require('moment');
const _ = require('lodash');
const projectId = 'xxx';
const bucketName = 'xxx';
const storage = new Storage({
projectId: projectId,
});
exports.uploadFile = (req, res) => {
if (req.method === 'POST') {
const busboy = new Busboy({
headers: req.headers
});
const uploads = []
const tmpdir = os.tmpdir();
busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
const filepath = path.join(tmpdir, filename)
var obj = {
path: filepath,
name: filename
}
uploads.push(obj);
var writeStream = fs.createWriteStream(obj.path);
file.pipe(writeStream);
});
busboy.on('finish', () => {
_.forEach(uploads, function (file) {
storage
.bucket(bucketName)
.upload(file.path, {
name: moment().format('/YYYY/MM/DD/x') + '-' + file.name
})
.then(() => {
console.log(`${file.name} uploaded to ${bucketName}.`);
})
.catch(err => {
console.error('ERROR:', err);
});
fs.unlinkSync(file.path);
})
res.end()
});
busboy.end(req.rawBody);
} else {
res.status(405).end();
}
}
Solved this with a stream instead of a temporary file. Only handles a single file at the moment though.
https://gist.github.com/PatrickHeneise/8f2c72c16c4e68e829e58ade64aba553#file-gcp-function-storage-file-stream-js
function asyncBusboy(req, res) {
return new Promise((resolve, reject) => {
const storage = new Storage()
const bucket = storage.bucket(process.env.BUCKET)
const fields = []
const busboy = Busboy({
headers: req.headers,
limits: {
fileSize: 10 * 1024 * 1024
}
})
busboy.on('field', (key, value) => {
fields[key] = value
})
busboy.on('file', (name, file, fileInfo) => {
const { mimeType } = fileInfo
const destFile = bucket.file(fileName)
const writeStream = destFile.createWriteStream({
metadata: {
contentType: fileInfo.mimeType,
metadata: {
originalFileName: fileInfo.filename
}
}
})
file.pipe(writeStream)
})
busboy.on('close', function () {
return resolve({ fields })
})
if (req.rawBody) {
busboy.end(req.rawBody)
} else {
req.pipe(busboy)
}
})
}