While developing on a 'clean' node.js frameworks, like Express.js - we can just easily handle file processing, callbacks, validations etc. But using Nest.js -> it's becoming simpler for simple issues, but more complex when you want to tune it...
I want to upload multiple files on the server, and right after that call another service action (like processing data from files async etc). Nest.js is using multer for this. And I got stuck...
import { Controller, HttpException, HttpStatus, Post, UploadedFiles, UseInterceptors } from '#nestjs/common';
import { AnyFilesInterceptor } from '#nestjs/platform-express';
import { diskStorage } from 'multer';
import { extname } from 'path';
import { v4 as uuid } from 'uuid';
#Controller('reader')
export class ReaderController {
#Post('upload')
#UseInterceptors(
AnyFilesInterceptor({
storage: diskStorage({
destination: './uploads',
fileSize: 5e7,
files: 20,
filename: (req, file, cb) => {
try {
const fileName = uuid();
return cb(null, `${fileName}${extname(file.originalname)}`);
} catch (err) {
return cb(new HttpException('Errored at upload', HttpStatus.BAD_REQUEST));
}
}
}),
fileFilter: (req: any, file: any, cb: any) => {
if (file.mimetype.match(/\/(png)$/)) {
cb(null, true);
} else {
cb(
new HttpException(
`Unsupported file type ${extname(file.originalname)}`,
HttpStatus.BAD_REQUEST
),
false
);
}
}
})
)
uploadFile(#UploadedFiles() files) {
console.log('files', files); // <--- here
}
}
How can I on my upload success action (or not success) handle it? How can i handle async file upload status? Like first file was uploaded with OK/NOK status and proceed with other files here (I will need to emit Socket action with 'file#n was uploaded with OK status')?
Normally I will just write something similar:
async.each(req.files, function(file, callback) {
saveFile(file, file.name, callback)
}, function(err) {
res.send('')
})
Related
I have a link of csv. So what i want is when my page load for the first time i want to download the csv file from a link and store it on public folder.So that i can take it from there and map its data by converting it into JSON.
First of all you need an express server. After that, create a file csv.js (maybe inside the "utils" folder) in your backend.
Inside write this code:
const {
readJSON,
writeJSON,
writeFile,
readFile,
remove,
createReadStream,
createWriteStream,
} = fs;
import fs from "fs-extra";
import { fileURLToPath } from "url";
import { dirname, join } from "path";
const authorsJSONPath = join(
dirname(fileURLToPath(import.meta.url)),
"../authors.json"
);
export const getAuthorsReadableStream = () => createReadStream(authorsJSONPath);
Then of course you need the server, and the routes. Import the router in your server and specify an endpoint.
If you know how to do that, create this route and import what you need. You will need the package json2csv
import json2csv from "json2csv";
// Get CSV files
router.get("/download/csv", async (req, res, next) => {
try {
res.setHeader("Content-Disposition", `attachment; filename=Authors.csv`);
const source = getAuthorsReadableStream();
const transform = new json2csv.Transform({
fields: ["id", "name", "surname", "email", "dateOfBirth", "avatar"],
});
const destination = res;
pipeline(source, transform, destination, (err) => {
if (err) next(err);
});
} catch (error) {
next(error);
}
});
If you want to just have that data to map through it, and not to be downloaded by the user, create a JSON file and map through it instead.
For any further support or issue just ask here.
DETAILS:
I have an express route which is set up to accept files as multipart/formdata and upload them to an S3 bucket. I am using multer to filter image types, as well as store them temporarily on the server through the creation of an upload directory. The files are removed shortly after upload success. The array of files are named images as per multer configuration, and accepts a maximum of 3 images.
The code works perfectly on my local machine. I test through POSTMAN and can upload 1-3 files and get the proper response. If there are no files attached, the correct response is triggered as well, all with status code 200.
PROBLEM:
The exact same codebase is deployed on Amazon ECS with Docker, but somehow keeps failing consistently with status code 500 and a generic 'Error' message that is not found in the codebase. Using logs I have determined that multer is not the cause, as it passes through the filter. It appears to be failing somewhere between the multer middleware and the route itself, with an exception.
Exception: Using POSTMAN, if a multipart/formdata POST request is made with no files I.E empty images array, the route is triggered properly and the message "You did not attach any images" is returned as a response.
I have been unable to figure out the issue and appreciate it if some guidance can be provided on this issue!
CODE SNIPPETS:
filesController:
files.post(
"/multiple",
upload.array("images", 3),
async (req: ExpressRequest, res: ExpressResponse) => {
try {
const files: { [fieldname: string]: Express.Multer.File[] } | Express.Multer.File[] =
req.files;
console.log("FILES", files);
// execute only if there are files
if (files.length > 0) {
const dataPromises = (files as Array<Express.Multer.File>).map(
async (file: Express.Multer.File) => {
// check if file.mimetype here is 'image/heic', and convert into jpeg accordingly
const fileNameWithoutExt = file.filename.split(".")[0];
try {
if (file.mimetype == "image/heic") {
await convertFile(file, fileNameWithoutExt, 0.2);
const response = await uploadFilePath(
S3_IMAGE_BUCKET,
`./uploads/${fileNameWithoutExt}.jpeg`,
`${fileNameWithoutExt}.jpeg`
);
console.log("HEIC File Upload Response", response);
fs.unlinkSync(`./uploads/${fileNameWithoutExt}.jpeg`);
fs.unlinkSync(file.path);
return {
fileName: `${fileNameWithoutExt}.jpeg`,
metaData: response.$metadata,
};
} else {
const response = await uploadFile(S3_IMAGE_BUCKET, file);
console.log("JPEG File Upload Response", response);
fs.unlinkSync(file.path);
return {
fileName: file.filename,
metaData: response.$metadata,
};
}
} catch (err) {
console.error("Error for file conversion/upload", err, err.stack);
res.status(500).send({
message: "Upload failed due to conversion or something.",
error: err,
stack: err.stack,
});
}
}
);
const fileData = await Promise.all(dataPromises);
const fileNames = fileData.map((data: any) => data.fileName);
const statusCodes = fileData.map((data: any) => data.metaData.httpStatusCode);
if (statusCodes.find((statusCode) => statusCode === 200)) {
res.status(200).send({
filePath: `/image/`,
fileNames,
});
} else {
res.status(403).send({
message: "Upload failed. Please check credentials or file has been selected.",
});
}
} else {
res.status(200).send({
message: "You did not attach any images",
});
}
} catch (err) {
res.status(500).send({
message: "Upload failed. Please check credentials or file has been selected.",
});
}
}
);
multer configuration:
const storage = multer.diskStorage({
// potential error, path to store files, callback
destination: (req, file, cb) => {
// cb acceptes two arguments: 1. err 2. destination folder wrt to server.js
cb(null, "uploads/");
},
filename: (req, file, cb) => {
console.log("MULTER STORAGE STARTED")
const date = new Date().toISOString().substring(0, 10);
// const name = `${req.body.first_name}_${req.body.last_name}`;
// cb defines the name of the file when stored
const alphabet = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz-";
const nanoid = customAlphabet(alphabet, 20);
cb(null, `${date}_${nanoid()}_${file.originalname}`);
console.log("FILE NAME CREATED, MULTER STORAGE STOPPED")
},
});
/* Accept jpeg or png files only */
// NOTE: file type rejection works, but there is no error message displayed if file is rejected. logic in route continues to be executed
const fileFilter = (
req: Request,
file: Express.Multer.File,
cb: (error: Error | null, accepted: boolean) => void
) => {
console.log("======== FILE FILTER ========", file);
if (
file.mimetype === "image/jpeg" ||
file.mimetype === "image/png" ||
file.mimetype === "image/heic"
) {
cb(null, true);
console.log("FILTER PASSED")
} else {
console.log("FILTER FAILED");
cb(null, false);
}
};
/* Only accepts filesize up to 5MB */
// the first parameter is super important that determines where the data is stored on the server
const upload = multer({
dest: "uploads/", // default simple config to upload file as binary data
storage, // enable if storage of actual file is required.
// limits: { fileSize: 1024 * 1024 * 5 },
fileFilter,
});
SCREENSHOTS:
response with no images in form data
response with images in form data
Can you make sure the upload directory exists in your Docker container? Multer will not create it if it doesn't exist. It might be failing silently between your storage function and the actual writing of the files to disk.
const storage = multer.diskStorage({
// potential error, path to store files, callback
destination: (req, file, cb) => {
// cb acceptes two arguments: 1. err 2. destination folder wrt to server.js
cb(null, "uploads/");
},
should be something like:
import { access, constants } from "fs";
import { join } from "path";
...
const storage = multer.diskStorage({
// potential error, path to store files, callback
destination: (req, file, cb) => {
// cb acceptes two arguments: 1. err 2. destination folder wrt to server.js
const currentDirectory: string = process.cwd();
const uploadDirectory: string = join(currentDirectory, 'uploads/');
// can we write to this path?
access(uploadDirectory, constants.W_OK, (err) => {
if (err) {
console.error(err);
cb(err, null);
}
cb(null, uploadDirectory);
})
},
I'm working (and learning) on my TypeScript skills, although I ran into a problem: I have a class named Manager which contains and manages multiple 'sub' managers. In the index file, I load the Manager by creating an instance and calling the load function. When loading all 'sub' managers get a reference to the main/only Manager instance, this way they can call/use the other 'sub' managers.
But I would like to be able to get some info from the 'sub' managers at a REST API endpoint. These endpoints are loaded through routes:
index.ts
import "reflect-metadata";
import { createConnection } from "typeorm";
import { Request, Response } from "express";
import * as express from "express";
import * as bodyParser from "body-parser";
import { AppRoutes } from "./routes";
import { Manager } from "./manager";
createConnection().then(async (typeORMConnection) => {
const manager = new Manager();
manager.load().then(() => {
console.log("Manager has loaded all managers");
const expressApp = express();
expressApp.use(bodyParser.json());
expressApp.use(function(req, res, next) {
res.header("Access-Control-Allow-Origin", "*");
res.header("Access-Control-Allow-Methods", "*");
res.header("Access-Control-Allow-Headers", "*");
next();
});
// Loop over every route
AppRoutes.forEach((singleRoute) => {
// Generate Express route
expressApp[singleRoute.method](singleRoute.path, (request: Request, response: Response, next: Function) => {
singleRoute.action(request, response)
.then(() => next())
.catch((error) => next(error));
});
});
// Start Express app
expressApp.listen(3000);
console.log("Express application is up and running on port 3000");
});
}).catch((error) => console.log(`TypeORM connection error: ${error}`));
A route file looks like this:
routes.ts
import { getSpeakerById, getSpeakerAll } from "./controller/get";
import { enableSpeakerById, disableSpeakerById } from "./controller/put";
export const AppRoutes = [
{
path: "/speaker",
method: "get",
action: getSpeakerAll
},
{
path: "/speaker/:id",
method: "get",
action: getSpeakerById
},
{
path: "/speaker/:id/disable",
method: "put",
action: disableSpeakerById
},
{
path: "/speaker/:id/enable",
method: "put",
action: enableSpeakerById
},
];
And last but not least this is an Express endpoint file containing the actual logic:
controller/get.ts
import { Request, Response } from "express";
import { getManager } from "typeorm";
import { Speaker } from "../entity/Speaker";
const ping = require("ping");
export async function getSpeakerById(request: Request, response: Response) {
const speakerRepository = getManager().getRepository(Speaker);
const speakerObject = await speakerRepository.findOne(request.params.id);
// If no post is found return 404
if (!speakerObject) {
response.status(404);
response.send("Speaker doesn't exist");
response.end();
return;
}
// Ping speaker and bind the time once its been resolved
speakerObject.time = await ping.promise.probe(speakerObject.host);
response.send(speakerObject);
}
export async function getSpeakerAll(request: Request, response: Response) {
const speakerRepository = getManager().getRepository(Speaker);
const speakerObjects = await speakerRepository.find();
const speakerPromise = [];
// Create a promise array of pings to all speakers
speakerObjects.forEach((speakerObject) => speakerPromise.push(ping.promise.probe(speakerObject.host)));
const speakerResults = await Promise.all(speakerPromise);
// Since the promise array is based on order we can rebind the property by looping over it in the same order
speakerResults.forEach((speakerResult, speakerIndex) => speakerObjects[speakerIndex].time = speakerResult.time);
response.send(speakerObjects);
}
Now I need to access the main Manager instance in the controller/get.ts, but I can't pass it along as parameter (for as far as I know) since it's an export. I would just import the Manager class and create a new instance but I only want to start the Manager once since it contains logic such as intervals and speaker instances from the Sonos package. I hope I was able to explain the problem but if anyone needs clarification on something I'll update the post.
You actually can pass it along as a parameter. There's nothing stopping you doing something like this in your index.ts:
// Generate Express route
expressApp[singleRoute.method](singleRoute.path, (request: Request, response: Response, next: Function) => {
singleRoute.action(request, response, manager)
.then(() => next())
.catch((error) => next(error));
});
});
And then updating the signature on you exported controller methods to be like
import { Manager } from '../manager';
export async function getSpeakerById(request: Request, response: Response, manager: Manager) {
...
}
I am trying to use the fastify-multer plugin to upload files to the server and I am able to successfully get the images uploaded to the folder.
The problem is my app crashes.
I used the fastify-cli generated structure and I am running it as a standalone server as mentioned in the README.md here.
I am writing it as a fastify plugin.
"use strict";
const fp = require("fastify-plugin");
module.exports = fp(function(fastify, opts, next) {
fastify.decorate("uploadImage", function(request) {
const path = require("path");
const multer = require("fastify-multer");
var storage = multer.diskStorage({
destination: path.join(path.join(__dirname + "/uploads/")),
filename: function(request, file, cb) {
cb(null, file.originalname);
}
});
var upload = multer({ storage }).single("myImage");
upload(request, function(err) {
if (err) {
console.log(err);
} else {
console.log("Saved...");
return { saved: true };
}
});
});
next();
});
And here is the error I get :
Hi looked into your issue. You are using fastify-multer in the wrong way.
Invoking multer({ storage }).single("myImage") you are creating a fastify's preHandler hook that accepts 3 specific parameters. You can find more on the offical documentation. A simple working example could be the one you can see at fastify-multer:
const server = fastify()
// register fastify content parser
server.register(multer.contentParser)
server.route({
method: 'POST',
url: '/profile',
preHandler: upload.single('avatar'),
handler: function(request, reply) {
// request.file is the `avatar` file
// request.body will hold the text fields, if there were any
reply.code(200).send('SUCCESS')
}
})
If you need more help just provide me a repro repo on github and I'll try to figure out what is the best solution for your case.
Let me know! :)
I am trying to send an image in a POST req from react to nodejs. However my back-end is not receiving the file and req.file is undefined. When I test the server image upload code with postman, everything works fine so I suspect something is wrong on the front-end. Anybody know what's wrong?
Here's my code:
Image_Upload.js (submit image func, fileList[0] is an image):
onSubmitImage = e => {
const config = {
headers: {
"content-type": "multipart/form-data"
}
};
const formData = new FormData();
formData.append(
"image",
this.state.fileList[0]
);
axios
.post("/api/profile/img_upload", formData)
.then(res => console.log("Response: " + res))
.catch(err => console.log(err.response.data));
};
profile.js (receiving image endpoint):
router.post(
"/img_upload",
passport.authenticate("jwt", { session: false }),
(req, res) => {
upload(req, res, err => {
console.log(req);
if (err) {
res.status(400).json({ Error: err });
} else {
if (req.file == undefined) {
res.status(404).json({ error: "no file selected" });
} else {
res.json({ fileLoc: req.file.location });
}
}
});
}
);
img_upload.js (multer setup):
const upload = multer({
storage: storage,
limits: { fileSize: 1000000 },
fileFilter: function(req, file, cb) {
checkFileType(file, cb);
}
}).single("image");
Any help is appreciated
EDIT:
I am noticing that when nodejs receives the req from the front-end, it is in req.body instead of req.file. So I am positive the problem is in react. How can I get react to send it as req.file instead of req.body?
Image request from Reacdt Front-end to Nodejs backend
upload that you defined is actually an express middleware which must take 3 arguments: request, response and next, respectively. next() calls the subsequent middleware in case no errors or next(error) otherwise. I highly recommend checking the answers to this question to get a better idea.
What you have done here is that you're calling a middleware as if it was a method. To fix that, config multer first
const multer = require('multer');
const upload = multer({
storage: 'path/here/',
limits: { fileSize: 1000000 },
fileFilter: function(req, file, cb) {
checkFileType(file, cb);
}
});
then
router.post("/img_upload",
passport.authenticate("jwt", { session: false }),
upload.single('image'),
(req, res) => {
// Your code here
});