I'm want to upload big file to my localhost server. And because the file size will be big ( 500MB maybe ), so I need to send the data chunk by chunk from client. Then my Nodejs server will keep appending the data to a temp file. Finally, I will change the file format and save it.
The problem is, I don't know how to send & read data properly. This is how I'm doing now, in my server I keep getting {} object.
Client Side:
// Prepare & Start Upload
const reader = new FileReader();
reader.addEventListener("load", async () => {
await fetch('/api/v1/file/upload/', {
method: 'POST',
body: reader.result
});
})
reader.readAsArrayBuffer( file.slice(0) ); // try to read whole file(small) first
My app setup:
/* Register Middlewares */
app.use(express.json())
app.use(express.urlencoded({extended:true}))
app.use('/api/v1/file/upload', fileUploadRouter)
app.use(express.static('./views'))
app.use(appError)
/* App Start */
const PORT = process.env.PORT || 3000
app.listen(PORT, () => console.log('Listening to: localhost:'+PORT))
My route
fileUploadRouter.post('/', async ( req, res, next ) => {
var buffer = req.body
console.log(buffer)
})
Thank for your help!
Related
I have a very basic question about a node application, and a question about HTTP requests. It's the first time I create a node app with server, and I just can't seem to get the different components to work together.
This is my server.js
var express = require('express');
var multer = require('multer');
const request = require('request');
const upload = multer({dest: __dirname + '/uploads/images'});
const app = express();
const PORT = 3000;
app.use(express.static('public'));
app.post('/upload', upload.single('photo'), (req, res) => {
if(req.file) {
res.json(req.file);
}
else throw 'error';
});
app.listen(PORT, () => {
console.log('Listening at ' + PORT );
});
Then I have a file app.js with a motion-detection system. Every time motion is detected, a picture is taken. This all works fine.
Then the picture should be sent to the server. This is what I can't figure out.
I created a function toServer() that should post the detected data to the server
const request = require('request');
function toServer(data) {
const formData = {
// Pass data via Buffers
my_buffer: data,
// Pass optional meta-data with an 'options' object with style: {value: DATA, options: OPTIONS}
// Use case: for some types of streams, you'll need to provide "file"-related information manually.
// See the `form-data` README for more information about options: https://github.com/form-data/form-data
};
request.post({url:'http://localhost:3000/upload', formData: formData}, function optionalCallback(err, httpResponse, body) {
if (err) {
return console.error('Upload failed:', err);
}
console.log('Upload successful! Server responded with:', body);
});
};
Problem 1: when running the server.js on localhost:3000, it doesn't find any of the scripts loaded in index.html nor my app.js.
Problem 2: when running the index.html on live-server, all scripts are found, but i get the error "request is not defined".
I am pretty sure there is some basic node setup thing I'm missing.
The solution for toServer() might be more complicated.
Thanks for your time,
Mustard Shaper
Problem 1:
this could happen because you have not specified to render your index.html.
for example:
res.render('index')
if it's not because of the single quotes in upload.single('photo') try double quotes.
Another possible error could be that you are missing a default display engine setting.
an example: https://www.npmjs.com/package/hbs
Problem 2:
it may be because you are missing the header
var request = require('request');
request.post({
headers: {'content-type' : 'application/x-www-form-urlencoded'},
url: 'http://localhost',
body: "example"
}, function(error, response, body){
console.log(body);
});
See more at https://expressjs.com/
I am using Express.js with Typescript and I would like to send a UInt8Array as binary data.
This is what I use so far and it works, but I would like not to save the file before, because I think it wastes performance:
const filePath = path.resolve(__dirname, 'template.docx');
const template = fs.readFileSync(filePath);
const buffer: Uint8Array = await createReport({
template,
data: {
productCode: data.productCode,
},
});
fs.writeFileSync(path.resolve(__dirname, 'output.docx'), buffer);
res.sendFile(path.resolve(__dirname, 'output.docx'));
I am using docx-templates to generate the file by the way.
You can use a PassThrough stream for this purpose, it'll keep the file in memory with no need to write to disk.
Something like this should do it:
const stream = require("stream");
const readStream = new stream.PassThrough();
// Pass your output.docx buffer to this
readStream.end(buffer);
res.set("Content-disposition", 'attachment; filename=' + "output.docx");
res.set("Content-Type", "application/vnd.openxmlformats-officedocument.wordprocessingml.document");
readStream.pipe(res);
The complete node.js code:
const fs = require("fs");
const express = require("express");
const port = 8000;
const app = express();
const stream = require("stream");
app.get('/download-file', (req, res) => {
const buffer = fs.readFileSync("./test.docx");
console.log("/download-file: Buffer length:", buffer.length);
const readStream = new stream.PassThrough();
readStream.end(buffer);
res.set("Content-disposition", 'attachment; filename=' + "test.docx");
res.set("Content-Type", "application/vnd.openxmlformats-officedocument.wordprocessingml.document");
readStream.pipe(res);
});
app.listen(port);
console.log(`Serving at http://localhost:${port}`);
To test, add a 'test.docx' file to the same directory, then point your browser to http://localhost:8000/download-file
Terry,
Thanks for the update of your answer and providing the full code. However, it still does not help much. I am trying to understand how I can handle this on the front-end side, in my case in Vue. Here is the following code:
router.post('/chart/word', async (req, res, next) => {
try {
if (!req.body.chartImage) throw new BadRequest('Missing the chart image from the request body')
const wordTemplate = await s3GetFile('folder', 'chart-templates-export/charts-template.docx')
const template = wordTemplate.Body
const buffer = await createReport({
cmdDelimiter: ["{", "}"],
template,
additionalJsContext: {
chart: () => {
const dataUrl = req.body.chartImage.src
const data = dataUrl.slice("data:image/jpeg;base64,".length);
return { width: 18 , height: 12, data, extension: '.jpeg' }
}
}
})
const stream = require('stream')
const readStream = new stream.PassThrough()
readStream.end(buffer)
res.set("Content-disposition", 'attachment; filename=' + "output.docx")
res.set("Content-Type", "application/vnd.openxmlformats-officedocument.wordprocessingml.document")
readStream.pipe(res)
} catch (err) {
console.log(err)
next(err)
}
})
And here is my Vue code, tested various stuff, but nothing...:
async exportCharts() {
console.log('this.$refs.test: ', this.$refs.test)
let img = {
src: this.$refs.test.getDataURL({
type: 'jpeg',
pixelRatio: window.devicePixelRatio || 1,
backgroundColor: '#fff'
}),
width: this.$refs.test.getWidth(),
height: this.$refs.test.getHeight()
}
const answersReq = await this.axios({
method: 'post',
url: '/pollAnswers/chart/word',
data: {
chartImage: img
}
responseType: 'arraybuffer' // 'blob' // 'document'
})
console.log('answersReq: ', answersReq)
if (answersReq.data) {
downloadURL(answersReq.data, 'report.docx')
}
}
What I am basically doing is: sending an image to the API (taken from html vue-echart element), then inserting it in a docx template, by using docx-templates library, which returns me Uint8Array that I want to export as the new Word Document with the populated charts. Then, the user (on the UI) should be able to choose the destination.
Here is the code for the download URL:
export function downloadURL(data, fileName) {
const mimeType = 'application/vnd.openxmlformats-officedocument.wordprocessingml.document'
const blob = new Blob([data], { type: mimeType })
const url = URL.createObjectURL(blob)
const element = document.createElement('a')
element.href = url
element.download = fileName
element.style.display = 'none'
document.body.appendChild(element)
element.click()
URL.revokeObjectURL(element.href)
document.body.removeChild(element)
}
P.S. Just to mention, if I directly save the buffer (the Uint8Array returned from the createReport) in the API, it works, the file is downloaded successfully and I can read it without any problems - it populates the correct chart in the file.
UPDATE:
I figured that out, but I am not sure why this is necessary and why it works that way and not the other. So, in the /chart/word endpoint, I am converting the Uint8Array buffer into a stream, then passing it as a response (the same way you used). Afterwards, in the Vue, I fetched this as responseType: 'arraybuffer', which converted the stream response into Uint8Array buffer again, then, I used the same method for the download and it works. Initially, I tried to send directly the buffer (without converting it as stream as you mentioned), but then on the front-end, the response was received as object that contained the Uint8Array buffer values, which was not what is expected and I could not create legit docx file. So, for some reason, it is required to convert the buffer as stream in the API, before sending it as response. Afterwards, on the front-end, I have to convert it back to arraybuffer and, finally, to make the docx download.
If you can explain to me why it works like that, I will be very happy.
I'm developing a web application using nodejs server-side. I'm trying to send pdf files from client to server.
Client:
var files = new FormData();
var count = 0;
$('#tableSlideId tr').each(function() {
var inputForm = $(this).find("th:first").children();
file = inputForm[0].files[0];
files.append((count++).toString(),file);
});
$.ajax({
type: "POST",
url: "/sendFiles",
data: files,
contentType: false,
processData: false,
}).done(function(err){
var text ="";
if(err) {
text = "Upload FAILED! Retry ...";
} else {
text = "Upload SUCCES!";
}
alert(text);
});
I think the client side is ok, infact if I use this loop:
for(var p of files)
console.log(p);
I correctly visualize all the elements that I want to send to the server.
Server:
app.post('/sendFiles', function(req,res) {
console.log("--->",req.body);
res.end();
});
Now in the server I have no idea how to visualize the data that I send, infact req.body is empty.
I don't know if this is the right way but my goal is to load some pdf files form the client, send to the server and after store them in a mySql dmbs.
Thank you.
Use express-formidable module. install 'express-formidable' by the running command
npm install express-formidable --save
a simple example is as follows from github
const express = require('express');
const formidable = require('express-formidable');
var app = express();
app.use(formidable());
app.post('/upload', (req, res) => {
//req.fields contains non-file fields
//req.files contains files
console.log(req.fields);
console.log(req.files);
});
Hope this helps!
Edit, from here -
app.post('/submit-form', (req, res) => {
new formidable.IncomingForm().parse(req, (err, fields, files) => {
if (err) {
console.error('Error', err)
throw err
}
console.log('Fields', fields)
console.log('Files', files)
files.map(file => {
console.log(file)
})
})
})
I think you need some middleware to accept the multipart formdata in the server side. Multer is a good option.
You can use
var multer = require('multer')
var upload = multer({ dest: 'uploads/' })
and then update your server side to handle the upload:
app.post('/sendFiles', upload.array('files', maxCount), function(req,res)
I have a file stored on an external server. I want to be able to call GET request to my own NodeJS server (using express). What I'm currently doing is almost OK, but it does not trigger browser to download the file (no browser UI for the download is shown):
const express = require('express');
const app = express();
app.get('/download-file', (req, res) => {
const externalRequest = http.request({
hostname: 'my.external-server.com',
path: '/my/path/my-file.zip',
}, (externalRes) => {
res.setHeader('Content-Disposition', 'attachment; filename="MyFile.zip"');
externalRes.pipe(res);
});
return externalRequest.end();
});
app.listen(8080, () => console.log('Server is listening'));
What am I missing here? I see that triggering a GET request to localhost:8080/download-file is actually fetching it, but no UI for download is shown.
This is the code that is running in one of my pet projects, hope it helps.
It pipes the download request ok, but there is no size info for the download, so it becames one of that downloads that you dont know when will finish.
const http = require('http')
app.get('/down_file/:file_name', (req, res) => {
const fileName = req.params.file_name
const url = "http://externalUrl/" + fileName
var externalReq = http.request(url, function(externalRes) {
res.setHeader("content-disposition", "attachment; filename=" + fileName);
externalRes.pipe(res);
});
externalReq.end();
})
I want to upload a file to sftp remote server using ssh2-sftp-client. I am taking the file from user in a post request along with destination. I am using multer to process the file.
const Client = require('ssh2-sftp-client');
const sftp = new Client();
const Multer = require("multer")
const multer = Multer({
storage: Multer.MemoryStorage
});
app.put("/sftp", multer.single('file'), (req, res) => {
sftpCredentials = req.query;
sftp.connect({
host: sftpCredentials.host,
port: sftpCredentials.port,
username: sftpCredentials.username,
password: sftpCredentials.password
}).then(res =>{
sftp.put(req.file,req.query.destination);
})
})
I am getting error :
TypeError: "string" must be a string, Buffer, or ArrayBuffer
sftp.put(localfilepath, remoteFilepath)
for localfilepath use:
req.file.path
You have used "req.file" only. If you want to get the filename too, use: req.file.originalname
Second, make sure "req.query.destination" is giving you the destination path where you want to put file.
And, do use of logging. It makes life easier.