I have a scenario that I need to upload an zip file.
In the zip file, there are lots of image files which will upload to AWS S3.
Because of the large amount of files in that zipfile, I want to get the information of upload process. In my opinion, the way I can get information is by using streaming response. Once server uploaded a file, respon a json to client.
Every time I upload a image to S3, I want to response a json object like the example bellow.
example for json streaming response:
{
"file_name": "imgae1.jpg",
"s3_url": "http://s3.url/key/to/file",
"other_key": "key for this uploaded file"
}
I'm trying to achieve this approach by using vue(cdn version) + axios(cdn version).
The code bellow which is how I upload my zip file.
function upload() {
var file = document.querySelector("#upload_file")
if (file.files.length <= 0) return
var formData = new FormData();
formData.append("file", file.files[0]);
formData.append("form_data", "form_data");
axios({
method: 'post',
url: "http://127.0.0.1:8000/",
headers: {
'Content-Type': 'multipart/form-data'
},
responseType: 'stream',
data: formData
}).then(function (response) {
if (response.status >= 200 && response.status < 300) {
alert("All images uploaded!")
}
})
}
but those examples I found are using axios npm package which I can't use.
Is there any recommend method or any resources that I can search?
Thanks for helping!
You can try using fetch instead like this:
fetch("http://example.url", {
method: "POST",
body: formData,
mode: "no-cors",
header: {
"Content-Type": "multipart/form-data",
},
}).then((response) => {
a = response.clone();
a.json().then((data) => {
//console.log('data', data)
});
});
Related
I have an API endpoint that lets the client post their csv to our server then post it to someone else server. I have done our server part which save uploaded file to our server, but I can't get the other part done. I keep getting error { message: 'File not found', code: 400 } which may mean the file never reach the server. I'm using axios as an agent, does anyone know how to get this done? Thanks.
// file = uploaded file
const form_data = new FormData();
form_data.append("file", fs.createReadStream(file.path));
const request_config = {
method: "post",
url: url,
headers: {
"Authorization": "Bearer " + access_token,
"Content-Type": "multipart/form-data"
},
data: form_data
};
return axios(request_config);
Update
As axios doc states as below and the API I'm trying to call requires a file
// data is the data to be sent as the request body
// Only applicable for request methods 'PUT', 'POST', and 'PATCH'
// When no transformRequest is set, must be of one of the following types:
// - string, plain object, ArrayBuffer, ArrayBufferView, URLSearchParams
// - Browser only: FormData, File, Blob
// - Node only: Stream, Buffer
Is there any way to make axios send a file as a whole? Thanks.
The 2 oldest answers did not work for me. This, however, did the trick:
const FormData = require('form-data'); // npm install --save form-data
const form = new FormData();
form.append('file', fs.createReadStream(file.path));
const request_config = {
headers: {
'Authorization': `Bearer ${access_token}`,
...form.getHeaders()
}
};
return axios.post(url, form, request_config);
form.getHeaders() returns an Object with the content-type as well as the boundary.
For example:
{ "content-type": "multipart/form-data; boundary=-------------------0123456789" }
I'm thinking the createReadStream is your issue because its async. try this.
Since createReadStream extends the event emitter, we can "listen" for when it finishes/ends.
var newFile = fs.createReadStream(file.path);
// personally I'd function out the inner body here and just call
// to the function and pass in the newFile
newFile.on('end', function() {
const form_data = new FormData();
form_data.append("file", newFile, "filename.ext");
const request_config = {
method: "post",
url: url,
headers: {
"Authorization": "Bearer " + access_token,
"Content-Type": "multipart/form-data"
},
data: form_data
};
return axios(request_config);
});
This is what you really need:
const form_data = new FormData();
form_data.append("file", fs.createReadStream(file.path));
const request_config = {
headers: {
"Authorization": "Bearer " + access_token,
"Content-Type": "multipart/form-data"
},
data: form_data
};
return axios
.post(url, form_data, request_config);
In my case, fs.createReadStream(file.path) did not work.
I had to use buffer instead.
const form = new FormData();
form.append('file', fs.readFileSync(filePath), fileName);
const config = {
headers: {
Authorization: `Bearer ${auth.access_token}`,
...form.getHeaders(),
},
};
axios.post(api, form.getBuffer(), config);
I have made an interceptor you can connect to axios to handle this case in node: axios-form-data. Any feedback would be welcome.
npm i axios-form-data
example:
import axiosFormData from 'axios-form-data';
import axios from 'axios';
// connect axiosFormData interceptor to axios
axios.interceptors.request.use(axiosFormData);
// send request with a file in it, it automatically becomes form-data
const response = await axios.request({
method: 'POST',
url: 'http://httpbin.org/post',
data: {
nonfile: 'Non-file value',
// if there is at least one streamable value, the interceptor wraps the data into FormData
file: createReadStream('somefile'),
},
});
// response should show "files" with file content, "form" with other values
// and multipart/form-data with random boundary as request header
console.log(response.data);
I had a same issue, I had a "pdf-creator-service" for generate PDF document from html.
I use mustache template engine for create HTML document - https://www.npmjs.com/package/mustache
Mustache.render function returns html as a string what do I need to do to pass it to the pdf-generator-service ? So lets see my suggestion bellow
//...
async function getPdfDoc(props: {foo: string, bar: string}): Promise<Buffer> {
const temlateFile = readFileSync(joinPath(process.cwd(), 'file.html'))
mustache.render(temlateFile, props)
const readableStream = this.getReadableStreamFromString(htmlString)
const formData = new FormData() // from 'form-data'
formData.append('file', options.file, { filename: options.fileName })
const formHeaders = formData.getHeaders()
return await axios.send<Buffer>(
{
method: 'POST',
url: 'https://pdf-generator-service-url/pdf',
data: formData,
headers: {
...formHeaders,
},
responseType: 'arraybuffer', // ! important
},
)
}
getReadableStreamFromString(str: string): Readable {
const bufferHtmlString = Buffer.from(str)
const readableStream = new Readable() // from 'stream'
readableStream._read = () => null // workaround error
readableStream.push(bufferHtmlString)
readableStream.push(null) // mark end of stream
return readableStream
}
For anyone who wants to upload files from their local filesystem (actually from anywhere with the right streams architecture) with axios and doesn't want to use any external packages (like form-data).
Just create a readable stream and plug it right into axios request function like so:
await axios.put(
url,
fs.createReadStream(path_to_file)
)
Axios accepts data argument of type Stream in node context.
Works fine for me at least in Node v.16.13.1 and with axios v.0.27.2
I'm trying to upload image to AWS S3 in my React Native(expo managed workflow), but in result the file is empty. I don't get any errors through the process. I've also tried to upload it using Uppy AWS plugin, but result is the same. Here is my code:
async function getUploadUrl(filename: string, type: string) {
const response = await fetch(GET_UPLOAD_URL(filename, type), {
method: 'GET',
headers: {
'Content-Type': 'application/json',
Accept: 'application/json',
},
});
return await response.json();
}
export default async function uploadImage(
file: Blob,
filename: string,
base64: string
) {
const uploadData = await getUploadUrl(filename, file.type);
const data = new FormData();
for (const [key, value] of Object.entries(uploadData.fields)) {
data.append(key, value as string);
}
data.append('file', Buffer.from(base64, 'base64'));
let res = await fetch(uploadData.url, {
method: 'post',
body: data,
headers: {
'Content-Type': 'multipart/form-data;',
},
});
I am using expo image picker to get the file. I've also tried to upload just Blob file insead of Buffer, but it doesn't work either
Here is how the file looks if i open it in browser https://prnt.sc/vOk5CI7lyPhu
If anyone also faced such problem, i managed to upload the file by uri like this:
formData.append('file', {
uri: uri,
type: 'image/jpeg',
name: filename,
});
I have an API endpoint that lets the client post their csv to our server then post it to someone else server. I have done our server part which save uploaded file to our server, but I can't get the other part done. I keep getting error { message: 'File not found', code: 400 } which may mean the file never reach the server. I'm using axios as an agent, does anyone know how to get this done? Thanks.
// file = uploaded file
const form_data = new FormData();
form_data.append("file", fs.createReadStream(file.path));
const request_config = {
method: "post",
url: url,
headers: {
"Authorization": "Bearer " + access_token,
"Content-Type": "multipart/form-data"
},
data: form_data
};
return axios(request_config);
Update
As axios doc states as below and the API I'm trying to call requires a file
// data is the data to be sent as the request body
// Only applicable for request methods 'PUT', 'POST', and 'PATCH'
// When no transformRequest is set, must be of one of the following types:
// - string, plain object, ArrayBuffer, ArrayBufferView, URLSearchParams
// - Browser only: FormData, File, Blob
// - Node only: Stream, Buffer
Is there any way to make axios send a file as a whole? Thanks.
The 2 oldest answers did not work for me. This, however, did the trick:
const FormData = require('form-data'); // npm install --save form-data
const form = new FormData();
form.append('file', fs.createReadStream(file.path));
const request_config = {
headers: {
'Authorization': `Bearer ${access_token}`,
...form.getHeaders()
}
};
return axios.post(url, form, request_config);
form.getHeaders() returns an Object with the content-type as well as the boundary.
For example:
{ "content-type": "multipart/form-data; boundary=-------------------0123456789" }
I'm thinking the createReadStream is your issue because its async. try this.
Since createReadStream extends the event emitter, we can "listen" for when it finishes/ends.
var newFile = fs.createReadStream(file.path);
// personally I'd function out the inner body here and just call
// to the function and pass in the newFile
newFile.on('end', function() {
const form_data = new FormData();
form_data.append("file", newFile, "filename.ext");
const request_config = {
method: "post",
url: url,
headers: {
"Authorization": "Bearer " + access_token,
"Content-Type": "multipart/form-data"
},
data: form_data
};
return axios(request_config);
});
This is what you really need:
const form_data = new FormData();
form_data.append("file", fs.createReadStream(file.path));
const request_config = {
headers: {
"Authorization": "Bearer " + access_token,
"Content-Type": "multipart/form-data"
},
data: form_data
};
return axios
.post(url, form_data, request_config);
In my case, fs.createReadStream(file.path) did not work.
I had to use buffer instead.
const form = new FormData();
form.append('file', fs.readFileSync(filePath), fileName);
const config = {
headers: {
Authorization: `Bearer ${auth.access_token}`,
...form.getHeaders(),
},
};
axios.post(api, form.getBuffer(), config);
I have made an interceptor you can connect to axios to handle this case in node: axios-form-data. Any feedback would be welcome.
npm i axios-form-data
example:
import axiosFormData from 'axios-form-data';
import axios from 'axios';
// connect axiosFormData interceptor to axios
axios.interceptors.request.use(axiosFormData);
// send request with a file in it, it automatically becomes form-data
const response = await axios.request({
method: 'POST',
url: 'http://httpbin.org/post',
data: {
nonfile: 'Non-file value',
// if there is at least one streamable value, the interceptor wraps the data into FormData
file: createReadStream('somefile'),
},
});
// response should show "files" with file content, "form" with other values
// and multipart/form-data with random boundary as request header
console.log(response.data);
I had a same issue, I had a "pdf-creator-service" for generate PDF document from html.
I use mustache template engine for create HTML document - https://www.npmjs.com/package/mustache
Mustache.render function returns html as a string what do I need to do to pass it to the pdf-generator-service ? So lets see my suggestion bellow
//...
async function getPdfDoc(props: {foo: string, bar: string}): Promise<Buffer> {
const temlateFile = readFileSync(joinPath(process.cwd(), 'file.html'))
mustache.render(temlateFile, props)
const readableStream = this.getReadableStreamFromString(htmlString)
const formData = new FormData() // from 'form-data'
formData.append('file', options.file, { filename: options.fileName })
const formHeaders = formData.getHeaders()
return await axios.send<Buffer>(
{
method: 'POST',
url: 'https://pdf-generator-service-url/pdf',
data: formData,
headers: {
...formHeaders,
},
responseType: 'arraybuffer', // ! important
},
)
}
getReadableStreamFromString(str: string): Readable {
const bufferHtmlString = Buffer.from(str)
const readableStream = new Readable() // from 'stream'
readableStream._read = () => null // workaround error
readableStream.push(bufferHtmlString)
readableStream.push(null) // mark end of stream
return readableStream
}
For anyone who wants to upload files from their local filesystem (actually from anywhere with the right streams architecture) with axios and doesn't want to use any external packages (like form-data).
Just create a readable stream and plug it right into axios request function like so:
await axios.put(
url,
fs.createReadStream(path_to_file)
)
Axios accepts data argument of type Stream in node context.
Works fine for me at least in Node v.16.13.1 and with axios v.0.27.2
When I am trying to upload a picture from the existing files it is working without any issues. But when I choose a file directly by clicking from the camera from mobile it fails showing "CORS Error". I added "Access-Control-Allow-Origin" and I removed it none of them helps.
userData.getRequestCallbackWithPrescription = (req, result) => {
var data = new FormData();
data.append("call_request[call_time]", req.call_time);
req.image_upload.forEach((obj, i) => {
data.append(`prescription_upload[image_upload][${i}][image]`, obj.file, obj.file.name)
})
return $http({
method: "POST",
url: globalUrl + '/api/' + apiVersion + '/customer/web/call_requests',
headers: {
"Access-Control-Allow-Origin": "*",
'Content-Type': undefined,
"X-Auth-Token": $rootScope.UserDetails.Auth,
},
data
})
.then(function (response) {
return result(response.data);
}, function (response) {
return result(response);
});
}
Please help me with finding the solution.
I just compressed the image and uploaded it. It works!
I am trying to upload a photo file to an S3 bucket using the Fetch API. I'm getting 400 Bad Request on the POST when trying to upload the photo. I am getting the presigned post url and the file details correctly but I believe the way I am formatting the formData is incorrect.
I'm using an html file input that uses onchange to run a javascript function handlePhoto.
The html is
<input type="file" onchange="handlePhoto()" id="file_input"/>
and javascript function is
function handlePhoto(){
const file = document.getElementById('file_input').files[0]
let formData = new FormData()
fetch("/v1/photos/get_presigned_post/" , {
method: "GET"
})
.then(response => response.json())
.then(s3Result => {
const { url, fields } = s3Result;
Object.keys(s3Result.fields).forEach(key => {
formData.append(key, s3Result.fields[key]);
});
formData.append('acl', 'public-read');
formData.append('Content-Type', file.type);
formData.append("file", file);
fetch(url, {
method: "POST",
body: formData,
headers: {
"Content-Type": "multipart/form-data"
}
})
});
}
Any help would be greatly appreciated.
Assuming you have a presigned url you could use something like this:
interface UploadIAToS3Params {
url: string;
file: File;
}
export function uploadToS3(params: UploadIAToS3Params) {
const { file, url } = params;
return fetch(url, { body: file, mode: "cors", method: "PUT" });
}
check the url in s3Result, for me it was wrong link and it doesn't include my bucketname. url must be like this to work:
https://{bucket_name}.s3.amazonaws.com
Your URL might not be right,
From the documentation, hostnames should look something like this.
"s3.Region.amazonaws.com".
These two links might help
https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAPI.html
and,
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTForms.html
You don't need to use a FormData, just pass the file with a content-type 'octet-stream'.
acl must be used when generating the presign URL.
fetch(url, {
method: 'PUT',
body: file,
headers: { 'Content-Type': 'application/octet-stream' }
})