I'm building a web application that requires to me to receive file binaries on a server-side endpoint, convert those binaries to FormData, then POST them to another endpoint. The second endpoint only accepts "multipart/form-data".
I'm using axios to create the requests, and NodeJS on the server.
The files are posted from the client as form data, but I'm unable to manipulate them as such once they reach the server.
How do I compel file binaries to FormData in my server-side scripts?
Client:
const handleImageSubmit = async (attachments) => {
try {
let image
if (attachments.length) {
const formData = new FormData()
for (let i = 0; i < attachments.length; i += 1) {
formData.append(
`files.${attachments[i].name}`,
attachments[i],
attachments[i].name
)
}
const config = {
headers: { 'Content-Type': 'multipart/form-data' },
}
image = await axios.post(
'/api/proxy/upload',
formData,
config
)
}
console.log('Files dispatched to server')
} catch (err) {
console.error(err)
}
}
Server-side:
const upload = async (req, res) => {
try {
const data = new FormData()
// Convert the binaries in req.body into FormData
const myFiles = ...
data.append('files',
myFiles)
const config = {
method: 'post',
url: `${process.env.API_URL}/upload`,
headers: {
Authorization: `Bearer ${req.headers['auth-token']}`,
...data.getHeaders()
},
data: data,
}
const signal = await axios(config)
res.json(signal.data)
} catch (err) {
console.error(err)
res.status(fetchResponse?.status || 500).json(err)
}
}
Related
I've been trying to send a JSON data using fetch but the backend receives an empty object.
In my Client JS I have
const user = "company1";
const username = "muneeb";
const data = {user, username};
fetch("http://127.0.0.1:3000/users/api/login", {
method: 'POST',
body: JSON.stringify(data)
}).then((response) => {
console.log(response);
});
The server side has:
router.post('/users/api/login', async (req, res, next) => {
try {
// console.log(request.body);
const request = JSON.stringify(req.body);
let imageTitles = [];
console.log(request);
*its random from here on out but you get the idea*
await components.getImages(imageTitles);
const finalKey = imageTitles.join("");
let images = await components.output(req.body.user ,req.body.username);
res.send(components.jsonConverter(imageTitles, images)); //---Top priority
db.setPassword(req.body.user, req.body.username , finalKey);
} catch (err) {
console.log(err);
res.send(err).sendStatus(500);
};
})
A few things I have already tried :
It works perfectly in Insomnia(postman).
express.json() is present , it helped me go from undefined to blank JSON.
I have enabled cors settings.
That's it for now.
The body parser express.json will only be applied for requests with Content-Type: application/json. You have to add the content type to your fetch call:
fetch("http://127.0.0.1:3000/users/api/login", {
method: 'POST',
body: JSON.stringify(data),
headers: {
'Content-Type': 'application/json'
}
})
I need to make this curl command into a nodejs request, either using fetch, request or axios networking libraries
The request needs to PUT the file data to the URL
curl -v -H "Content-Type:application/octet-stream" --upload-file C:/Users/deanv/Pictures/test3.mp4 "https://www.example.com/file/upload"
I've tried using CurlConverter to convert my curl command to node code, but this doesn't work as I cant add the file data.
var fetch = require('node-fetch');
fetch('https://www.example.com/file/upload', {
method: 'PUT',
headers: {
'Content-Type': 'application/octet-stream'
}
});
All help appreciated. Thanks in advance.
Try this:
// setup modules
const fetch = require('node-fetch');
const FormData = require('form-data');
const fs = require('fs');
const path = require('path');
// setup paths
const pathToFile = 'C:/Users/deanv/Pictures/test3.mp4';
const uploadUrl = 'https://www.example.com/file/upload';
// create form, 'content-type' header will be multipart/form-data
const form = new FormData();
// read the file as stream
const fileStream = fs.createReadStream(path.resolve(pathToFile));
// add the file to the form
form.append('my_file', fileStream);
fetch(uploadUrl, {
method: 'PUT',
body: form
}).then((res) => {
console.log('done: ', res.status);
return res.text();
})
.then((res) => {
console.log('raw response: ', res);
})
.catch(err => {
console.log('err', err);
});
I am using an existing API call to send a file to our cloud provider via Nodejs. I have seen several different methods of doing this online, but figured I would stick to using "fetch" as most of my other API calls have been using this as well. Presently, I keep getting 500 internal server error and am not sure why? My best conclusion is that I am not sending the file properly or one of my pieces of formdata are not resolving correctly. See the below code:
const fetch = require("node-fetch");
const formData = require("form-data");
const fs = require("fs");
var filePath = "PATH TO MY FILE ON SERVER WITH FILE NAME";
var accessToken = "Bearer <ACCESS TOKEN>;
var url = '<API URL TO CLOUD PROVIDER>';
var headers = {
'Content-Type': 'multipart/form-data',
'Accept': 'application/json',
'Authorization': accessToken
};
const form = new formData();
const buffer = fs.readFileSync(filePath);
const apiName = "MY_FILE_NAME";
form.append("Content-Type", "application/octect-stream");
form.append("file", filePath);
console.log(form);
fetch(url, { method: 'POST', headers: headers, body: form })
.then(response => response.json())
.then(data => {
console.log(data)
})
.catch(err => {
console.log(err)
});
This my first time attempting something like this so I am next to certain I am missing something. Any help with getting me in the right direction is appreciated.
So the issue was exactly what I mentioned above. The code was not uploading the file I specified. I finally figured out why and below is the modified code which will grab the file and upload to our cloud service provide:
const fetch = require("node-fetch");
const formData = require("form-data");
const fs = require("fs");
var apiName = process.env['API_PATH'];
var accessToken = "Bearer" +" "+ process.env['BEARER_TOKEN'];
var url = process.env['apiEnv'] +"/" +"archive";
var headers = {
'Accept': 'application/json',
'Authorization': accessToken,
};
const form = new formData();
const buffer = fs.readFileSync(apiName);
const uploadAPI = function uploadAPI() {
form.append("Content-Type", "application/octet-stream");
form.append('file', buffer);
fetch(url, {method: 'POST', headers: headers, body: form})
.then(data => {
console.log(data)
})
.catch(err => {
console.log(err)
});
};
uploadAPI();
Being new to Javascript/Nodejs I wasn't really sure what the "buffer" variable did. After finally figuring it out I realized I was adding too many body form params to the request and the file was not being picked up and sent to the provider. All code above is using custom variables, but if for whatever reason someone wants to use it, then simply replace the custom variables with your own....Thanks again for any and all assistance....
import fs from 'fs'
import FormData from 'FormData';
const fileStream = fs.createReadStream('./file.zip');
const form = new FormData();
form.append('key', fileStream, 'file.zip');
const response = await axios.post(url, form, {
headers: {
...form.getHeaders(),
},
});
Im trying to read a file (image) from amazon S3 and post it to another server with multipart/form.
let imageParams = { Bucket: 'my-bucket', Key: 'imageName.jpg' };
let formData = new FormData();
formData.append('file', s3.getObject(imageParams).createReadStream());
let apiResponse = await api.post("/fileUpload", formData,
{ params: { token: process.env.API_TOKEN } },
{ headers: {'Content-Type': 'multipart/form-data' } } );
But im not managing it to work, it returns me:
Error: Request failed with status code 415
maybe im misunderstanding how the createReadStream() works?
Use concat for pipe the stream. Otherwise form data send only the first chunk of stream, and the server don't know how to handle it.
For example
const axios = require('axios');
const {S3} = require('aws-sdk');
const FormData = require('form-data');
const s3 = new S3();
var concat = require('concat-stream')
const api = axios.default.create({
baseURL: 'http://example.com',
})
const readStream = s3.getObject({Bucket: 'bucket', Key: 'file'}).createReadStream();
readStream.pipe(concat(filebuffer => {
const formData = new FormData();
formData.append('file', filebuffer);
formData.getLength((err, length) => {
console.log(length)
const headers = formData.getHeaders({'content-length': length})
console.log({headers})
return api({
method: 'post',
url: "/upload",
headers: headers,
data: formData
})
})
}))
I am using "Axios" to call a WCF method that takes as parameter file information and content.
The file is read and sent as a base64 encoded string.
My issue is that when the file size exceeds a certain limit, AXIOS throws an exception: "Error: Request body larger than maxBodyLength limit".
I looked up the issue and found that all solutions suggest increasing the maxContentLength / maxBodyLength parameters in the AXIOS configuration object, but did not succeed.
Find Below an implemented test case in node.js:
var axios = require('axios');
var fs = require('fs');
var path = require('path')
var util = require('util')
let readfile = util.promisify(fs.readFile)
async function sendData(url,data) {
let params = data
let resp = await axios({
method: 'post',
url: url,
data: JSON.stringify(params),
headers: { 'Accept': 'application/json', 'Content-Type': 'application/json' }
// maxContentLength: 100000000,
// maxBodyLength: 1000000000
}).catch(err => {
throw err;
})
return resp;
}
async function ReadFile(filepath) {
try{
let res = await readfile(filepath,'base64')
let filename = path.basename(filepath).split('.').slice(0, -1).join('.')
let ext = path.extname(filepath)
return {data:res,fext:ext,fname:filename}
let x = 1
}
catch(err)
{
throw err
}
}
(async () => {
try {
let img = await ReadFile('Files/1.pdf')
let res = await sendData('http://183.183.183.242/EMREngineEA/EMRWS.svc/web/EMR_TestUploadImg',img)
console.log(res)
}
catch (ex) {
console.log(ex)
}
}
)();
In my case, the pdf file is 20 MB, upon running, an error is thrown.
"Error: Request body larger than maxBodyLength limit"
I tried to setting the maxContentLength: 100000000, maxBodyLength: 1000000000
as presented above, but did not succeed.
Your help is appreciated.
The maxBodyLength seems to work for me in this simple test, I upload data to a local Express server. If I try to upload more than the maxBodyLength I get the same error you're getting. So I suspect there's something more, like a redirect happening in your case that's triggering the error.
There is an issue logged for axios here that seems to reference the problem, it suggests setting maxContentLength to Infinity (as the other commenter suggests).
e.g.
maxContentLength: Infinity,
maxBodyLength: Infinity
Test code below:
const axios = require("axios");
function generateRandomData(size) {
const a = Array.from({length: size}, (v, k) => Math.floor(Math.random()*100));
return { data: a, id: 1 };
}
async function uploadData(url, size) {
let params = generateRandomData(size);
let stringData = JSON.stringify(params);
console.log(`uploadData: Uploading ${stringData.length} byte(s)..`);
let resp = await axios({
method: 'post',
url: url,
data: stringData,
headers: { 'Accept': 'application/json', 'Content-Type': 'application/json' },
maxContentLength: 100000000,
maxBodyLength: 1000000000
}).catch(err => {
throw err;
})
console.log("uploadData: response:", resp.data);
return resp;
}
uploadData("http://localhost:8080/upload", 10000000);
Corresponding server code:
const express = require("express");
const port = 8080;
const app = express();
const bodyParser = require('body-parser')
app.use(bodyParser.json({limit: '50mb'}));
app.post('/upload', (req, res, next) => {
console.log("/upload: Received data: body length: ", req.headers['content-length']);
res.json( { status: 'ok', bytesReceived: req.headers['content-length']});
})
app.listen(port);
console.log(`Serving at http://localhost:${port}`);