I'm attempting to create a server-side upload component in node.js, but I'm having trouble interpreting the information sent from PLUpload. From what I can tell, PLUpload (in HTML5 mode) sends files as binary information, which creates problems for the node.js packages I've been attempting to use so far (node-formidable and node-express), as they expect normal HTML uploads with multipart content types.
For what it's worth, this is the code I've been attempting to use...
var formidable = require('formidable');
var sys = require('sys');
http.createServer( function( req, res ){
console.log('request detected');
if( req.url == '/upload/' ){
console.log('request processing');
var form = new formidable.IncomingForm();
form.parse( req, function( err, fields, files ){
res.writeHead( 200, {
'Access-Control-Allow-Origin': 'http://tksync.com',
'Access-Control-Allow-Methods': 'GET,PUT,POST,DELETE',
'Access-Control-Allow-Headers': '*',
'content-type': 'text/plain'
});
res.write('received upload:\n');
res.end(sys.inspect({
fields: fields,
files: files
}));
});
}
}).listen( 8080 );
I have no problem to use plupload(in HTML5 mode) with node.js with below code:
module.exports.savePhoto= (req, res) ->
if req.url is "/upload" and req.method.toLowerCase() is "post"
console.log 'savePhoto: req.url=', req.url, 'req.method=', req.method
form = new formidable.IncomingForm()
files = []
fields = []
form.uploadDir = config.PATH_upload
form.on("field", (field, value) ->
console.log field, value
fields.push [ field, value ]
).on("file", (field, file) ->
console.log field, file
files.push [ field, file ]
).on "end", ->
console.log "-> upload done: fields=", fields
console.log "received fields:", util.inspect(fields)
console.log "received files:", util.inspect(files)
size = files[0][1].size
pathList = files[0][1].path.split("/")
#console.log 'pathList=', pathList
file = pathList[pathList.length - 1]
console.log "file=" + file
......
I created node-pluploader to handle this, as I found elife's answer didn't work for chunked uploads, as said chunks come in on different requests.
Express-based usage example:
var Pluploader = require('node-pluploader');
var pluploader = new Pluploader({
uploadLimit: 16
});
/*
* Emitted when an entire file has been uploaded.
*
* #param file {Object} An object containing the uploaded file's name, type, buffered data & size
* #param req {Request} The request that carried in the final chunk
*/
pluploader.on('fileUploaded', function(file, req) {
console.log(file);
});
/*
* Emitted when an error occurs
*
* #param error {Error} The error
*/
pluploader.on('error', function(error) {
throw error;
});
// This example assumes you're using Express
app.post('/upload', function(req, res){
pluploader.handleRequest(req, res);
});
Related
DETAILS:
I have an express route which is set up to accept files as multipart/formdata and upload them to an S3 bucket. I am using multer to filter image types, as well as store them temporarily on the server through the creation of an upload directory. The files are removed shortly after upload success. The array of files are named images as per multer configuration, and accepts a maximum of 3 images.
The code works perfectly on my local machine. I test through POSTMAN and can upload 1-3 files and get the proper response. If there are no files attached, the correct response is triggered as well, all with status code 200.
PROBLEM:
The exact same codebase is deployed on Amazon ECS with Docker, but somehow keeps failing consistently with status code 500 and a generic 'Error' message that is not found in the codebase. Using logs I have determined that multer is not the cause, as it passes through the filter. It appears to be failing somewhere between the multer middleware and the route itself, with an exception.
Exception: Using POSTMAN, if a multipart/formdata POST request is made with no files I.E empty images array, the route is triggered properly and the message "You did not attach any images" is returned as a response.
I have been unable to figure out the issue and appreciate it if some guidance can be provided on this issue!
CODE SNIPPETS:
filesController:
files.post(
"/multiple",
upload.array("images", 3),
async (req: ExpressRequest, res: ExpressResponse) => {
try {
const files: { [fieldname: string]: Express.Multer.File[] } | Express.Multer.File[] =
req.files;
console.log("FILES", files);
// execute only if there are files
if (files.length > 0) {
const dataPromises = (files as Array<Express.Multer.File>).map(
async (file: Express.Multer.File) => {
// check if file.mimetype here is 'image/heic', and convert into jpeg accordingly
const fileNameWithoutExt = file.filename.split(".")[0];
try {
if (file.mimetype == "image/heic") {
await convertFile(file, fileNameWithoutExt, 0.2);
const response = await uploadFilePath(
S3_IMAGE_BUCKET,
`./uploads/${fileNameWithoutExt}.jpeg`,
`${fileNameWithoutExt}.jpeg`
);
console.log("HEIC File Upload Response", response);
fs.unlinkSync(`./uploads/${fileNameWithoutExt}.jpeg`);
fs.unlinkSync(file.path);
return {
fileName: `${fileNameWithoutExt}.jpeg`,
metaData: response.$metadata,
};
} else {
const response = await uploadFile(S3_IMAGE_BUCKET, file);
console.log("JPEG File Upload Response", response);
fs.unlinkSync(file.path);
return {
fileName: file.filename,
metaData: response.$metadata,
};
}
} catch (err) {
console.error("Error for file conversion/upload", err, err.stack);
res.status(500).send({
message: "Upload failed due to conversion or something.",
error: err,
stack: err.stack,
});
}
}
);
const fileData = await Promise.all(dataPromises);
const fileNames = fileData.map((data: any) => data.fileName);
const statusCodes = fileData.map((data: any) => data.metaData.httpStatusCode);
if (statusCodes.find((statusCode) => statusCode === 200)) {
res.status(200).send({
filePath: `/image/`,
fileNames,
});
} else {
res.status(403).send({
message: "Upload failed. Please check credentials or file has been selected.",
});
}
} else {
res.status(200).send({
message: "You did not attach any images",
});
}
} catch (err) {
res.status(500).send({
message: "Upload failed. Please check credentials or file has been selected.",
});
}
}
);
multer configuration:
const storage = multer.diskStorage({
// potential error, path to store files, callback
destination: (req, file, cb) => {
// cb acceptes two arguments: 1. err 2. destination folder wrt to server.js
cb(null, "uploads/");
},
filename: (req, file, cb) => {
console.log("MULTER STORAGE STARTED")
const date = new Date().toISOString().substring(0, 10);
// const name = `${req.body.first_name}_${req.body.last_name}`;
// cb defines the name of the file when stored
const alphabet = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz-";
const nanoid = customAlphabet(alphabet, 20);
cb(null, `${date}_${nanoid()}_${file.originalname}`);
console.log("FILE NAME CREATED, MULTER STORAGE STOPPED")
},
});
/* Accept jpeg or png files only */
// NOTE: file type rejection works, but there is no error message displayed if file is rejected. logic in route continues to be executed
const fileFilter = (
req: Request,
file: Express.Multer.File,
cb: (error: Error | null, accepted: boolean) => void
) => {
console.log("======== FILE FILTER ========", file);
if (
file.mimetype === "image/jpeg" ||
file.mimetype === "image/png" ||
file.mimetype === "image/heic"
) {
cb(null, true);
console.log("FILTER PASSED")
} else {
console.log("FILTER FAILED");
cb(null, false);
}
};
/* Only accepts filesize up to 5MB */
// the first parameter is super important that determines where the data is stored on the server
const upload = multer({
dest: "uploads/", // default simple config to upload file as binary data
storage, // enable if storage of actual file is required.
// limits: { fileSize: 1024 * 1024 * 5 },
fileFilter,
});
SCREENSHOTS:
response with no images in form data
response with images in form data
Can you make sure the upload directory exists in your Docker container? Multer will not create it if it doesn't exist. It might be failing silently between your storage function and the actual writing of the files to disk.
const storage = multer.diskStorage({
// potential error, path to store files, callback
destination: (req, file, cb) => {
// cb acceptes two arguments: 1. err 2. destination folder wrt to server.js
cb(null, "uploads/");
},
should be something like:
import { access, constants } from "fs";
import { join } from "path";
...
const storage = multer.diskStorage({
// potential error, path to store files, callback
destination: (req, file, cb) => {
// cb acceptes two arguments: 1. err 2. destination folder wrt to server.js
const currentDirectory: string = process.cwd();
const uploadDirectory: string = join(currentDirectory, 'uploads/');
// can we write to this path?
access(uploadDirectory, constants.W_OK, (err) => {
if (err) {
console.error(err);
cb(err, null);
}
cb(null, uploadDirectory);
})
},
I have a very basic question about a node application, and a question about HTTP requests. It's the first time I create a node app with server, and I just can't seem to get the different components to work together.
This is my server.js
var express = require('express');
var multer = require('multer');
const request = require('request');
const upload = multer({dest: __dirname + '/uploads/images'});
const app = express();
const PORT = 3000;
app.use(express.static('public'));
app.post('/upload', upload.single('photo'), (req, res) => {
if(req.file) {
res.json(req.file);
}
else throw 'error';
});
app.listen(PORT, () => {
console.log('Listening at ' + PORT );
});
Then I have a file app.js with a motion-detection system. Every time motion is detected, a picture is taken. This all works fine.
Then the picture should be sent to the server. This is what I can't figure out.
I created a function toServer() that should post the detected data to the server
const request = require('request');
function toServer(data) {
const formData = {
// Pass data via Buffers
my_buffer: data,
// Pass optional meta-data with an 'options' object with style: {value: DATA, options: OPTIONS}
// Use case: for some types of streams, you'll need to provide "file"-related information manually.
// See the `form-data` README for more information about options: https://github.com/form-data/form-data
};
request.post({url:'http://localhost:3000/upload', formData: formData}, function optionalCallback(err, httpResponse, body) {
if (err) {
return console.error('Upload failed:', err);
}
console.log('Upload successful! Server responded with:', body);
});
};
Problem 1: when running the server.js on localhost:3000, it doesn't find any of the scripts loaded in index.html nor my app.js.
Problem 2: when running the index.html on live-server, all scripts are found, but i get the error "request is not defined".
I am pretty sure there is some basic node setup thing I'm missing.
The solution for toServer() might be more complicated.
Thanks for your time,
Mustard Shaper
Problem 1:
this could happen because you have not specified to render your index.html.
for example:
res.render('index')
if it's not because of the single quotes in upload.single('photo') try double quotes.
Another possible error could be that you are missing a default display engine setting.
an example: https://www.npmjs.com/package/hbs
Problem 2:
it may be because you are missing the header
var request = require('request');
request.post({
headers: {'content-type' : 'application/x-www-form-urlencoded'},
url: 'http://localhost',
body: "example"
}, function(error, response, body){
console.log(body);
});
See more at https://expressjs.com/
trying to learn D3 I wrote the following local server:
const http = require('http');
const fs = require('fs');
function onRequest(request, response) {
response.writeHead(200, {'Content-Type': 'text/html'});
fs.readFile('./index.html', null, function(error, data) {
if (error) {
response.writeHead(404);
// response.write('file not found');
} else {
response.write(data);
}
response.end();
});
}
http.createServer(onRequest).listen(8000, '127.0.0.1');
I then go to http://127.0.0.1:8000/ to render this index.html:
<html>
<body>
<script src="https://d3js.org/d3.v5.min.js"></script>
<script>
var stringit = `[{"coin_val": 4999999,"coin_lab": "#PAX"},{"coin_val": 1100000,"coin_lab": "#USDC"}]`;
console.log('working.')
d3.json('./data.json', function(err, data) {
console.log(err)
console.log(data)
});
</script>
</body>
</html>
but I receive the following error in Chrome console:
Uncaught (in promise) SyntaxError: Unexpected token < in JSON at
position 1
at Go (d3.v5.min.js:2) Go # d3.v5.min.js:2
What am I doing wrong? is it i the 3D code or I just don't get the server right? how can I get D3 to read a JSON file in a Node.js server?
I suspect the JSON is not the issue, something goes wrong on the server side and reads the HTML in a wrong way?
I wrote the following local server
Which serves up the contents of index.html in response to any request it gets.
d3.json('./data.json',
So your JavaScript asks for data.json and it gets the contents of index.html.
Since the contents of index.html are not JSON, and start with a <, it throws the error message. A JSON file cannot start with a <.
You need to fix your server so it gives the browser what it asks for instead of blindly sending index.html.
your problem seems to be that your code doesn't know how to serve anything but index.html. It is really frustrating working with a pure node server because most resources on the internet assume that users are going to employ express or another framework.
Below I have a server that can serve static websites and handle requests for a few common types of media. You can add other types by modifying the code in the getContentType function by looking up the mime type for that file format.
I hope this helps
'use strict'
// Step 1: Declare Constants and Require External Resources
const port = "8888", landing = 'index.html', hostname = "127.0.0.1";
const path = require('path');
const http = require('http');
const fs = require('fs');
const qs = require('querystring');
// Step 2: Create getContentType Function that Returns the Requested MimeType for the browser
/**
* getContentType :: str -> str
*
* Function returns the content type that matches the resource being
* requested by the server controller
*/
function getContentType(url){
const mimeTypes = {
'.html' : 'text/html' , '.js' : 'text/javascript' ,
'.css' : 'text/css' , '.json' : 'application/json' ,
'.png' : 'image/png' , '.jpg' : 'image/jpg' ,
'.gif' : 'image/gif' , '.svg' : 'image/svg+xml' ,
'.wav' : 'audio/wav' , '.mp4' : 'video/mp4' ,
'.woff' : 'application/font-woff' , '.ttf' : 'application/font-ttf' ,
'.otf' : 'application/font-otf' , '.eot' : 'application/vnd.ms-fontobject' ,
'.wasm' : 'application/wasm'
};
// If requested url extension is a mime type, the dict object will return that url's value,
// otherwise octet-stream will be returned instead
return mimeTypes[path.extname(url).toLowerCase()] || 'application/octet-stream';
}
// Step 3: Create sendFile Function that Delivers Requested files to the Response stream
/**
* sendFile :: (str, str, str, stream) -> void
*
* function delivers any requested resources to the stream
*/
function sendFile(file, url, contentType, request, response){
fs.readFile(file, (error, content) => {
if(error) {
response.writeHead(404)
.write(`404 Error: '${url}' Was Not Found!`);
response.end();
// include file path for easy debugging, tabs added to make distinct
console.log(`\t${request.method} Response: 404 Error, '${file}' Was Not Found!`);
} else {
response.writeHead(200, {'Content-Type': contentType})
.write(content);
response.end();
console.log(`\t${request.method} Response: 200, ${url} Served`);
};
});
};
// Step 4: Create serverController Function to initialize the server and run the request loop
/**
* serverController :: str -> void
*
* Function creates a server and accesses sendFile and getContentType to serve
* requested resources
*/
function serverController(hostname) {
const server = http.createServer((request, response) => {
// Creates space around .html requests so that they stand out more in the console
if (path.extname(request.url) == '.html' || request.url == '/') {
console.log(`\nPage Requested: ${request.url}\n`);
} else {
if (request.method == "GET") {
console.log(`${request.method} Request: ${request.url}`);
} else {
console.log(`Request came: ${request.url}`);
}
}
// Sends the requested resources to the response stream
if (request.url == '/') {
var file = path.join(__dirname, landing); // delivers index.html by default
sendFile(file, landing, 'text/html', request, response);
} else {
var file = path.join(__dirname, request.url); // delivers requested resource
sendFile(file, request.url, getContentType(request.url), request, response);
};
});
// Gives server a port to listen to and gives an IP address to find it
server.listen(port, hostname, () => {
console.log(`Server running at ${hostname}:${port}\n`);
});
}
// Step 6: Create launch IIFE Function that Starts the server upon Instantiation
(function launch() {
serverController(hostname);
})();
So I'm running into an issue with ExpressJS and can't seem to find documentation to solve the issue.
Tech:
body-parser: 1.17.0
express 4.15.0
multer: 1.3.0
MongoDB
Postman
View is currently 3 fields:
name (required)
tagline (required)
image (optional)
What I'm trying to do is Error Handle on the Image before writing anything into the database. The image can only be of mime type image/jpeg or image/png to prevent HTML being uploaded with malicious JS inside.
I believe the issue seems to be that I'm not properly triggering an error while running through the image check conditionals and sending multiple responses which is setting off the Error: Can't set headers after they are sent.
drinks.routes.js
var express = require('express');
var router = express.Router();
var jwt = require('jsonwebtoken');
var multer = require('multer');
var passport = require('passport');
var config = require('../config/main');
var upload = multer({ dest: 'uploads/images' })
var Drink = require('../models/drinks.model.js');
router.use(function(req, res, next){
next();
});
...
.post(passport.authenticate('jwt', { session: false }), upload.single('image'), function(err, req, res, next){
var drink = req.body;
var drinkImage = req.file;
if(typeof drinkImage !== "undefined"){
console.log('image was uploaded');
if(drinkImage.mimetype !== "image/jpeg" || drinkImage.mimetype !== "image/png" ){
console.log('Image was not a JPEG or PNG', drinkImage.mimetype);
res.status(500).send({ error: "Your image was incorrect"}); // >>>>>>>>>>>>>>> The error seems to be coming from here. Unsure of how to properly raise a flag to tell the response to the client. Have tried res.send(), the res.status().send(), res.json(), currently working with next() method to keep going on but not sure how to define err if that is the case
}
console.log('image correct mimetype');
} else {
drinkImage = {}; // Setting this as an empty object so it doesn't throw an error with the model which is looking for `image: drinkImage.name`
}
Drink.createDrink(drink, drinkImage, function(err, drink, drinkImage){
if(err){
console.log('Error adding Drink', err);
res.send(err);
}
res.status(200).json(drink)
});
});
Topic Research
Express Error Handling
SO Can't Set Headers Twice
This issue is due to Javascript's asynchronous nature.
The code should not execute Drink.createDrink() after responding error 500.
if (drinkImage.mimetype !== "image/jpeg" || drinkImage.mimetype !== "image/png" ) {
console.log('Image was not a JPEG or PNG', drinkImage.mimetype);
res.status(500).send({ error: "Your image was incorrect"});
return; // THIS IS VERY IMPORTANT!
}
I have file from client that i have to read on server side and send back to client for download , How can i acheive that task using nodejs. I tried with fs but i am getting some error.
console.log(data) is coming as empty object
server.js
var multiparty = require('multiparty');
var data = new multiparty.Form();
export function create(req, res) {
data.parse(req, function(err, fields, files) {
console.log(files);
var fileContent = fs.readFileSync(files.file[0].path,'utf8');
res.json(fileContent );
});
}
router.js
var express = require('express');
var controller = require('./fileUpload.controller');
var router = express.Router();
router.post('/fileUpload',controller.create);
module.exports = router;
fileData
{ file:
[ { fieldName: 'file',
originalFilename: 'sco_poc.bpmn',
path: 'C:\\Users\\9u\\AppData\\Local\\Temp\\f4DG8L7nCpNyNvVPYqGPkd44.bpmn',
headers: [Object],
size: 11078 } ] }
I am assuming you are trying download a local file the path from your JSON object 'fileData'. My example below is written in NodeJS
First, you will need to stringify your JSONobject
var jsonString = JSON.stringify({ file:
[ { fieldName: 'file',
originalFilename: 'sco_poc.bpmn',
path: 'C:\\Users\\9u\\AppData\\Local\\Temp\\f4DG8L7nCpNyNvVPYqGPkd44.bpmn',
headers: [Object],
size: 11078 } ] });
//console.log(jsonString)//print jsonString contents
Second, parse it into a JavaScript object
var jsonObj = JSON.parse(jsonString);
//console.log(jsonObj); //print jsonObj contents
Third, get path from jsonObj
var path = jsonObj.file[0].path;
Finally, read the (local) file
fs.readFile(path,function(err,data){
var fileData="";
fileData+=data;
res.writeHead(200, {
'Location': '<if needed>',
'Content-Type':'<expected content-type>'
});
res.end(fileData); //ends response, and sends to client
});
If you look at the very first example on the multiparty NPM page here: https://www.npmjs.com/package/multiparty, you will see that you need to run this for each new request, not just once that you reuse over and over:
var form = new multiparty.Form();
So, for starter move that into your request handler. Then, if you're unsure how to use the results, I'd suggest you add this:
console.log(fields, files);
And, this should show you what data you actually have.
FYI, you can see errors in the parsing with this:
form.on('error', function(err) {
console.log('Error parsing form: ' + err.stack);
});
Also, note this statement from the documentation:
If cb is provided, autoFields and autoFiles are set to true and all
fields and files are collected and passed to the callback, removing
the need to listen to any events on form. This is for convenience when
you want to read everything, but be sure to write cleanup code, as
this will write all uploaded files to the disk, even ones you may not
be interested in.
You will need to cleanup files on disk after each request or they will accumulate.