I want to upload a file to sftp remote server using ssh2-sftp-client. I am taking the file from user in a post request along with destination. I am using multer to process the file.
const Client = require('ssh2-sftp-client');
const sftp = new Client();
const Multer = require("multer")
const multer = Multer({
storage: Multer.MemoryStorage
});
app.put("/sftp", multer.single('file'), (req, res) => {
sftpCredentials = req.query;
sftp.connect({
host: sftpCredentials.host,
port: sftpCredentials.port,
username: sftpCredentials.username,
password: sftpCredentials.password
}).then(res =>{
sftp.put(req.file,req.query.destination);
})
})
I am getting error :
TypeError: "string" must be a string, Buffer, or ArrayBuffer
sftp.put(localfilepath, remoteFilepath)
for localfilepath use:
req.file.path
You have used "req.file" only. If you want to get the filename too, use: req.file.originalname
Second, make sure "req.query.destination" is giving you the destination path where you want to put file.
And, do use of logging. It makes life easier.
Related
I have uploaded the file in my backend filesystem using multer
My server is node and client is react.
I'm having trouble downloading and displaying the saved file on the client react
Whenever I do res.download(file) it just throws an error as connection refused on client side.
My code is as follows:
UserToUploadMapping.js
const mongoose = require("mongoose");
const UserToUploadMapping = new mongoose.Schema({
userId: {
type:String,
required:true
},
file: {
type: Object,
required: true,
},
date: {
type: Date,
default: Date.now,
},
});
module.exports = mongoose.model("UserToUploadMapping", UserToUploadMapping);
uploadVideo.js
const router = require("express").Router();
const multer = require('multer');
const UserToUploadMapping = require('../models/UserToUploadMapping')
let nameFile = ''
const storage = multer.diskStorage({
destination:'./Videos',
filename:(req,file,cb) => {
console.log(file)
nameFile = file.originalname + " "+ Date.now()
cb(null, nameFile)
}
})
const upload = multer({storage:storage})
router.post('/upload', upload.single('video'), async (req,res,next) => {
console.log("object")
const saveMapping = new UserToUploadMapping({
userId:'123',
file:req.file,
})
await saveMapping.save()
res.send("Video uploaded")
})
router.get('/download', async(req,res,next) => {
const x = await UserToUploadMapping.find()
// res.send(x)
res.download(x[0].path)
})
module.exports = router;
CLIENT
const fetchVideo = async () => {
const resp = await axios.get(
"http://localhost:5000/api/user/video/download"
);
console.log(resp)
};
return (
<>
<NavContainer />
<div className={classes.Post}>
<Input
type="file"
onChange={(e) => uploadVideos(e.target.files)}
accept="video/mp4"
/>
{/* <Button onClick={(e) => submitHandler(e)}>Upload</Button> */}
<video></video>
</div>
</>
);
Error
There is a few problems within the uploadVideo.js file :
to get the path from the data, you need to use x[0].file.path
(based on how you save the file in the database)
const saveMapping = new UserToUploadMapping({
userId:'123',
file:req.file,
})
to avoid problems about where the file uploadVideo.js is and where we run the application, you should use an absolute path when saving files in the system.
(small problem) your filename function will give filenames like this video.mp4 1622180824748. I think this is better "video-1622181268053.mp4" (we have the correct file extension)
You can refer to this code
const router = require("express").Router();
const multer = require('multer');
const UserToUploadMapping = require('../models/UserToUploadMapping')
const path = require('path');
const uploadFolder = path.join(__dirname, "Videos"); // use a variable to hold the value of upload folder
const storage = multer.diskStorage({
destination: uploadFolder, // use it when upload
filename: (req, file, cb) => {
// nameFile = file.originalname + " "+ Date.now() // --> give "video.mp4 1622180824748"
let [filename, extension] = file.originalname.split('.');
let nameFile = filename + "-" + Date.now() + "." + extension; // --> give "video-1622181268053.mp4"
cb(null, nameFile)
}
})
const upload = multer({ storage: storage })
router.post('/upload', upload.single('video'), async (req, res, next) => {
const saveMapping = new UserToUploadMapping({
userId: '123',
file: req.file,
})
await saveMapping.save()
res.send("Video uploaded")
})
router.get('/download', async (req, res, next) => {
const video = await UserToUploadMapping.find({});
res.download(video[0].file.path); // video[0].file.path is the absolute path to the file
})
module.exports = router;
Your code indicates you are handling large files (videos). I would strongly recommend looking at separation of concerns, handling this as part of your other business logic is not recommended based on my experience. This can e.g. complicate firewall rules and DDOS protection when that is needed in the future.
As a minimum, move upload and download into its own server, e.g. 'files.yourappnamehere.com' so that you can handle the specifics separately from your business logic api.
If you run in the public cloud, I would strongly recommend looking at reusing blob upload/download functionality, letting your clients upload directly to blob storage and also handling downloads directly from blob storage, e.g. in Azure, AWS or GCP.
This will save you a lot of the implementation details of handling (very) large files, and also give "free" extensibility options such as events on file upload completion.
You are running 2 apps Frontend and Backend with difference ports (3000, 5000) so browsers block cross domain requests. On Express you must enable CORS to allow request from FrontEnd Url (http://localhost:3000).
For the download route, try using window.location functions instead of using Axios.
It looks like you might have a typo in your get handler... you're referencing an element called 'path', but that's not declared in your schema
router.get('/download', async(req,res,next) => {
const x = await UserToUploadMapping.find()
// res.send(x)
res.download(x[0].path)//<-Path Doesn't seem to be in the schema
})
Since you don't have a try/catch in that function, the resulting error could be bringing down your server, making it unavailable
You might also want to take a look at this for more detail on How to download files using axios
good evening.
I'm trying to create a POST request with a file and some data on a REST API I'm building using NodeJS.
If not clear, my goal to this feature of the API is to save a register of a picture, so I'd like to send the picture file, the picture name and it's number on the same request.
I'm currently using Jest / supertest for testing and to test this specific functionality, I've tried the following:
const response = await request(app)
.post("/uploads/pics")
.field("name", "PicureName")
.field("number", "PictureNumber")
.attach("file", picture);
I've read this from https://visionmedia.github.io/superagent/#multipart-requests
My problem is that I can't get the values of name and number on my request on my controller, so I can't use them to save the object.
I've tried many ways, such as:
req.body.name
req.name
req.field.name
req.query.name
but none of these worked for me.
I also tried printing the whole request, however I couldn't find anything related to name, number or field there.
Does anyone can tell what I'm doing wrong ?
You should use https://github.com/expressjs/multer middleware for handling file upload. Then, req.body will hold the text fields, if there were any.
E.g.
index.js:
const express = require('express');
const multer = require('multer');
const app = express();
const upload = multer({ dest: 'uploads/' });
app.post('/uploads/pics', upload.single('file'), (req, res) => {
console.log(req.body);
console.log(req.file);
res.sendStatus(200);
});
module.exports = app;
index.test.js:
const request = require('supertest');
const app = require('./');
const path = require('path');
const { expect } = require('chai');
describe('62862866', () => {
it('should pass', async () => {
const picture = path.resolve(__dirname, './test.jpg');
const response = await request(app)
.post('/uploads/pics')
.field('name', 'PicureName')
.field('number', 'PictureNumber')
.attach('file', picture);
expect(response.status).to.be.eq(200);
});
});
integration test result:
62862866
[Object: null prototype] { name: 'PicureName', number: 'PictureNumber' }
{ fieldname: 'file',
originalname: 'test.jpg',
encoding: '7bit',
mimetype: 'image/jpeg',
destination: 'uploads/',
filename: '181b96eb9044aac5d50c8c1e3159a120',
path: 'uploads/181b96eb9044aac5d50c8c1e3159a120',
size: 0 }
✓ should pass (84ms)
1 passing (103ms)
I'm developing a web application using nodejs server-side. I'm trying to send pdf files from client to server.
Client:
var files = new FormData();
var count = 0;
$('#tableSlideId tr').each(function() {
var inputForm = $(this).find("th:first").children();
file = inputForm[0].files[0];
files.append((count++).toString(),file);
});
$.ajax({
type: "POST",
url: "/sendFiles",
data: files,
contentType: false,
processData: false,
}).done(function(err){
var text ="";
if(err) {
text = "Upload FAILED! Retry ...";
} else {
text = "Upload SUCCES!";
}
alert(text);
});
I think the client side is ok, infact if I use this loop:
for(var p of files)
console.log(p);
I correctly visualize all the elements that I want to send to the server.
Server:
app.post('/sendFiles', function(req,res) {
console.log("--->",req.body);
res.end();
});
Now in the server I have no idea how to visualize the data that I send, infact req.body is empty.
I don't know if this is the right way but my goal is to load some pdf files form the client, send to the server and after store them in a mySql dmbs.
Thank you.
Use express-formidable module. install 'express-formidable' by the running command
npm install express-formidable --save
a simple example is as follows from github
const express = require('express');
const formidable = require('express-formidable');
var app = express();
app.use(formidable());
app.post('/upload', (req, res) => {
//req.fields contains non-file fields
//req.files contains files
console.log(req.fields);
console.log(req.files);
});
Hope this helps!
Edit, from here -
app.post('/submit-form', (req, res) => {
new formidable.IncomingForm().parse(req, (err, fields, files) => {
if (err) {
console.error('Error', err)
throw err
}
console.log('Fields', fields)
console.log('Files', files)
files.map(file => {
console.log(file)
})
})
})
I think you need some middleware to accept the multipart formdata in the server side. Multer is a good option.
You can use
var multer = require('multer')
var upload = multer({ dest: 'uploads/' })
and then update your server side to handle the upload:
app.post('/sendFiles', upload.array('files', maxCount), function(req,res)
I have a file stored on an external server. I want to be able to call GET request to my own NodeJS server (using express). What I'm currently doing is almost OK, but it does not trigger browser to download the file (no browser UI for the download is shown):
const express = require('express');
const app = express();
app.get('/download-file', (req, res) => {
const externalRequest = http.request({
hostname: 'my.external-server.com',
path: '/my/path/my-file.zip',
}, (externalRes) => {
res.setHeader('Content-Disposition', 'attachment; filename="MyFile.zip"');
externalRes.pipe(res);
});
return externalRequest.end();
});
app.listen(8080, () => console.log('Server is listening'));
What am I missing here? I see that triggering a GET request to localhost:8080/download-file is actually fetching it, but no UI for download is shown.
This is the code that is running in one of my pet projects, hope it helps.
It pipes the download request ok, but there is no size info for the download, so it becames one of that downloads that you dont know when will finish.
const http = require('http')
app.get('/down_file/:file_name', (req, res) => {
const fileName = req.params.file_name
const url = "http://externalUrl/" + fileName
var externalReq = http.request(url, function(externalRes) {
res.setHeader("content-disposition", "attachment; filename=" + fileName);
externalRes.pipe(res);
});
externalReq.end();
})
I'm trying to create and then send zip file to client. I know how to create it but I've got a problem with send it to client. I tried many ways.
I'm sending POST request from Client and as response I want to send a file.
This is my server-site example code
var Zip = require('node-zip');
router.post('/generator', function(req, res, next) {
var zip = new Zip;
zip.file('hello.txt', 'Hello, World!');
var options = {base64: false, compression:'DEFLATE'};
fs.writeFile('test1.zip', zip.generate(options), 'binary', function (error) {
console.log('wrote test1.zip', error);
});
res.setHeader('Content-disposition', 'attachment; filename=test1.zip');
res.download('test1.zip');
}
});
I also tried something like this:
res.setHeader('Content-disposition', 'attachment; filename=' + filename);
res.setHeader('Content-type', mimetype);
var filestream = fs.createReadStream(file);
filestream.pipe(res);
I tried to use such libraries as:
node-zip
archiver
Can anyone explain me how to do that ?
This module works fine too: https://www.npmjs.com/package/adm-zip
Example without creating temporary zip file in server:
var AdmZip = require('adm-zip');
router.get('/zipFilesAndSend', function(req, res) {
var zip = new AdmZip();
// add local file
zip.addLocalFile("./uploads/29/0046.xml");
// get everything as a buffer
var zipFileContents = zip.toBuffer();
const fileName = 'uploads.zip';
const fileType = 'application/zip';
res.writeHead(200, {
'Content-Disposition': `attachment; filename="${fileName}"`,
'Content-Type': fileType,
})
return res.end(zipFileContents);
});
Try this express-easy-zip npm package to generate a zip file from a local folder path and send it as a download to the client.
var zip = require('express-easy-zip');
var app = require('express')();
app.use(zip());
app.get('my-route/zip', async function(req, res) {
var dirPath = __dirname + "/uploads";
await res.zip({
files: [{
path: dirPath,
name: 'Package'
}],
filename: 'Package.zip'
});
});
I haven't worked with node-zip or archiver before (I usually just use the built-in zlib module), but one thing I noticed right away is that you should place res.download inside the callback of writeFile. That way it will only send the file once it has been fully written to disk.
fs.writeFile('test1.zip', zip.generate(options), 'binary', function (error) {
res.download('test1.zip');
});
I hope this solution works for you, if it doesn't feel free to comment.
Also, I think res.download sets the Content-disposition header for you, you don't need to set it manually. Not 100% sure on that one though.
Above solutions work.(above solutions generate zip and send it to frontend as data in response. In order to make it as downloadable following code will work) I was using express-zip. It is compressing files and sending data to frontend from backend(node). But in frontend I was getting only data in response. In my case I want user can be able to download the zip which sent by server. To solve this I followed following approach. For generating download window in browser i used downloadjs (we can follow another approach but i find this easy)
Front-End
const download = require('downloadjs')
return axios({
url:process.env.API_HOST+'/getuploadedfiles',
method:'get',
headers:{
'Content-Type': 'multipart/form-data',
withCredentials:true,
},
responseType:'arraybuffer' // If we don't mention we can't get data in desired format
})
.then(async response => {
console.log("got al files in api ");
let blob = await new Blob([response.data], { type: 'application/zip' }) //It is optional
download(response.data,"attachement.zip","application/zip") //this is third party it will prompt download window in browser.
return response.data;
})
Backe-End
const zip = require('express-zip');
app.use('/getuploadedfiles',function(req,res){
res.zip([
{path:'/path/to/file/file2.PNG',name:'bond.png'},
{path:'/path/to/file/file1.PNG',name:'james.png'}
])