i want download a pdf file with axios and save on disk (server side) with fs.writeFile, i have tried:
axios.get('https://xxx/my.pdf', {responseType: 'blob'}).then(response => {
fs.writeFile('/temp/my.pdf', response.data, (err) => {
if (err) throw err;
console.log('The file has been saved!');
});
});
the file is saved but the content is broken...
how do I correctly save the file?
Actually, I believe the previously accepted answer has some flaws, as it will not handle the writestream properly, so if you call "then()" after Axios has given you the response, you will end up having a partially downloaded file.
This is a more appropriate solution when downloading slightly larger files:
export async function downloadFile(fileUrl: string, outputLocationPath: string) {
const writer = createWriteStream(outputLocationPath);
return Axios({
method: 'get',
url: fileUrl,
responseType: 'stream',
}).then(response => {
//ensure that the user can call `then()` only when the file has
//been downloaded entirely.
return new Promise((resolve, reject) => {
response.data.pipe(writer);
let error = null;
writer.on('error', err => {
error = err;
writer.close();
reject(err);
});
writer.on('close', () => {
if (!error) {
resolve(true);
}
//no need to call the reject here, as it will have been called in the
//'error' stream;
});
});
});
}
This way, you can call downloadFile(), call then() on the returned promise, and making sure that the downloaded file will have completed processing.
Or, if you use a more modern version of NodeJS, you can try this instead:
import * as stream from 'stream';
import { promisify } from 'util';
const finished = promisify(stream.finished);
export async function downloadFile(fileUrl: string, outputLocationPath: string): Promise<any> {
const writer = createWriteStream(outputLocationPath);
return Axios({
method: 'get',
url: fileUrl,
responseType: 'stream',
}).then(response => {
response.data.pipe(writer);
return finished(writer); //this is a Promise
});
}
You can simply use response.data.pipe and fs.createWriteStream to pipe response to file
axios({
method: "get",
url: "https://xxx/my.pdf",
responseType: "stream"
}).then(function (response) {
response.data.pipe(fs.createWriteStream("/temp/my.pdf"));
});
The problem with broken file is because of backpressuring in node streams. You may find this link useful to read: https://nodejs.org/es/docs/guides/backpressuring-in-streams/
I'm not really a fan of using Promise base declarative objects in JS codes as I feel it pollutes the actual core logic & makes the code hard to read. On top of it, you have to provision event handlers & listeners to make sure the code is completed.
A more cleaner approach on the same logic which the accepted answer proposes is given below. It uses the concepts of stream pipelines.
const util = require('util');
const stream = require('stream');
const pipeline = util.promisify(stream.pipeline);
const downloadFile = async () => {
try {
const request = await axios.get('https://xxx/my.pdf', {
responseType: 'stream',
});
await pipeline(request.data, fs.createWriteStream('/temp/my.pdf'));
console.log('download pdf pipeline successful');
} catch (error) {
console.error('download pdf pipeline failed', error);
}
}
exports.downloadFile = downloadFile
I hope you find this useful.
// This works perfectly well!
const axios = require('axios');
axios.get('http://www.sclance.com/pngs/png-file-download/png_file_download_1057991.png', {responseType: "stream"} )
.then(response => {
// Saving file to working directory
response.data.pipe(fs.createWriteStream("todays_picture.png"));
})
.catch(error => {
console.log(error);
});
The following code taken from https://gist.github.com/senthilmpro/072f5e69bdef4baffc8442c7e696f4eb?permalink_comment_id=3620639#gistcomment-3620639 worked for me
const res = await axios.get(url, { responseType: 'arraybuffer' });
fs.writeFileSync(downloadDestination, res.data);
node fileSystem writeFile encodes data by default to UTF8. which could be a problem in your case.
Try setting your encoding to null and skip encoding the received data:
fs.writeFile('/temp/my.pdf', response.data, {encoding: null}, (err) => {...}
you can also decalre encoding as a string (instead of options object) if you only declare encoding and no other options. string will be handled as encoding value. as such:
fs.writeFile('/temp/my.pdf', response.data, 'null', (err) => {...}
more read in fileSystem API write_file
I have tried, and I'm sure that using response.data.pipe and fs.createWriteStream can work.
Besides, I want to add my situation and solution
Situation:
using koa to develop a node.js server
using axios to get a pdf via url
using pdf-parse to parse the pdf
extract some information of pdf and return it as json to browser
Solution:
const Koa = require('koa');
const app = new Koa();
const axios = require('axios')
const fs = require("fs")
const pdf = require('pdf-parse');
const utils = require('./utils')
app.listen(process.env.PORT || 3000)
app.use(async (ctx, next) => {
let url = 'https://path/name.pdf'
let resp = await axios({
url: encodeURI(url),
responseType: 'arraybuffer'
})
let data = await pdf(resp.data)
ctx.body = {
phone: utils.getPhone(data.text),
email: utils.getEmail(data.text),
}
})
In this solution, it doesn't need to write file and read file, it's more efficient.
This is what worked for me and it also creates a temporary file for the image file in case the output file path is not specified:
const fs = require('fs')
const axios = require('axios').default
const tmp = require('tmp');
const downloadFile = async (fileUrl, outputLocationPath) => {
if(!outputLocationPath) {
outputLocationPath = tmp.fileSync({ mode: 0o644, prefix: 'kuzzle-listener-', postfix: '.jpg' });
}
let path = typeof outputLocationPath === 'object' ? outputLocationPath.name : outputLocationPath
const writer = fs.createWriteStream(path)
const response = await axios.get(fileUrl, { responseType: 'arraybuffer' })
return new Promise((resolve, reject) => {
if(response.data instanceof Buffer) {
writer.write(response.data)
resolve(outputLocationPath.name)
} else {
response.data.pipe(writer)
let error = null
writer.on('error', err => {
error = err
writer.close()
reject(err)
})
writer.on('close', () => {
if (!error) {
resolve(outputLocationPath.name)
}
})
}
})
}
Here is a very simple Jest test:
it('when downloadFile should downloaded', () => {
downloadFile('https://i.ytimg.com/vi/HhpbzPMCKDc/hq720.jpg').then((file) => {
console.log('file', file)
expect(file).toBeTruthy()
expect(file.length).toBeGreaterThan(10)
})
})
Lorenzo's response is probably the best answer as it's using the axios built-in.
Here's a simple way to do it if you just want the buffer:
const downloadFile = url => axios({ url, responseType: 'stream' })
.then(({ data }) => {
const buff = []
data.on('data', chunk => buff.push(chunk))
return new Promise((resolve, reject) => {
data.on('error', reject)
data.on('close', () => resolve(Buffer.concat(buff)))
})
})
// better
const downloadFile = url => axios({ url, responseType: 'arraybuffer' }).then(res => res.data)
const res = await downloadFile(url)
fs.writeFileSync(downloadDestination, res)
I'd still probably use the 'arraybuffer' responseType
There is a much simpler way that can be accomplished in a couple of lines:
const fileResponse = await axios({
url: fileUrl,
method: "GET",
responseType: "stream",
});
// Write file to disk (here I use fs.promise but you can use writeFileSync it's equal
await fsPromises.writeFile(filePath, fileResponse.data);
Axios has internal capacity of handling streams and you don't need to necessarily meddle with low-level Node APIs for that.
Check out https://axios-http.com/docs/req_config (find the responseTypepart in the docs for all the types you can use).
If you just want the file use this
const media_data =await axios({url: url, method: "get", responseType: "arraybuffer"})
writeFile("./image.jpg", Buffer.from(media_data.data), {encoding: "binary"}, console.log)
import download from "downloadjs";
export const downloadFile = async (fileName) => {
axios({
method: "get",
url: `/api/v1/users/resume/${fileName}`,
responseType: "blob",
}).then(function (response) {
download(response.data, fileName);
});
};
it's work fine to me
This is my example code run with node js
There is a synctax error
should be writeFile not WriteFile
const axios = require('axios');
const fs = require('fs');
axios.get('http://www.africau.edu/images/default/sample.pdf', {responseType: 'blob'}).then(response => {
fs.writeFile('./my.pdf', response.data, (err) => {
if (err) throw err;
console.log('The file has been saved!');
});
});
After the file is saved it might look like in a text editor, but the file was saved properly
%PDF-1.3
%����
1 0 obj
<<
/Type /Catalog
/Outlines 2 0 R
/Pages 3 0 R
>>
endobj
2 0 obj
<<
/Type /Outlines
/Count 0
>>
endobj
3 0 obj
<<
/Type /Pages
/Count 2
/Kids [ 4 0 R 6 0 R ]
>>
endobj
I'm working on an backend API but at some point I need to get user data from another API. I am trying to use Axios to make http request in order to do that. The request return the result in the browser as expected but the problem is that I can't display console log in the terminal. It doesn't show anything even though I asked the program to do so. Is there a problem probably with my code?
Here is my code :
const axios = require('axios');
const AxiosLogger = require('axios-logger');
const instance = axios.create();
module.exports = (router) => {
router.get('/profile', function(req, res) {
//random fake profile info
axios.get('https://randomuser.me/api/')
.then(response => {
console.log(response.data);
console.log(response.data);
return response.data
})
.catch(error => {
console.log(error);
});
});
};
I would suggest trying response.send to forward the axios response to your client like so:
module.exports = (router) => {
router.get('/profile', function(req, res) {
//random fake profile info
axios.get('https://randomuser.me/api/')
.then(response => {
console.log(response.data);
// Send the axios response to the client...
res.send(response.data)
})
.catch(error => {
console.log(error);
});
});
};
I am working with shopify admin api. I create a bulk operation and in return it gives me a url. When you hit this url it automatically downloads the file. This file as information that my server needs.
So where I am at - I send the request for bulk operation, then I hit the url to download file and file gets downloaded by my browser.
What I need is to hit the url, and then download and save the file into a folder on my server where I can then start working with it. And I dont know how to do this, I tried, I looked around. Any help would be awesome. Thanks.
My code:
app.get("/postBulkProds", async (req,res) => {
let url = GRAPHQL_CUSTOM_URL;
await fetch(url, postBulkProducts())
.then(res=> res.json())
.then(bulk => res.send(bulk))
.catch(error => res.send(error));
});
app.get("/getBulkProds", async(req,res, next) => {
let data;
let url = GRAPHQL_CUSTOM_URL;
await fetch(url, getBulkProducts())
.then(products=> (products.json(products)))
.then(products => (data = products))
.catch(error => res.send(error));
console.log(data.data.currentBulkOperation.url);
res.locals.bulk = data;
let url2 = data.data.currentBulkOperation.url;
res.redirect(url2);
});
EDIT-SOLVED
fetch(url2)
.then(r => r.text())
//.then( t=> console.log(t))
.then(t => {let json=JSON.stringify(t);
//console.log(json);
fs.writeFile('./myfile.jsonl', json, err => {
if(err){
console.log("error writing", err);
} else {
console.log("success");
}
})
})
.catch(error => console.log(error));
I have a Node Js server, in it, I am fetching a blob data from another web service (which is for a PDF file), now after receiving blob, I want to convert it again into PDF file.
Anyone, who knows how to achieve this please help.
Here is my code block I have tried so far:
const fetch = require('node-fetch');
const Blob = require('fetch-blob');
const fs = require('fs');
fetch(url, options)
.then(res => {
console.log(res);
res.blob().then(async (data) => {
const result = data.stream();
// below line of code saves a blank pdf file
fs.createWriteStream(objectId + '.pdf').write(result);
})
})
.catch(e => {
console.log(e);
});
Modification points:
For fs.createWriteStream(objectId + '.pdf').write(data), please modify res.blob() to res.buffer().
Please modify .then(res => {res.blob().then() to .then(res => res.buffer()).then(.
Modified script:
fetch(url, options)
.then(res => res.buffer())
.then(data => {
fs.createWriteStream(objectId + '.pdf').write(data);
})
.catch(e => {
console.log(e);
});
Note:
In this modification, it supposes that the fetch process using url and options works fine.
References:
node-fetch
write()
Using Node.js, I am trying to get an image from a URL and upload that image to another service without saving image to disk. I have the following code that works when saving the file to disk and using fs to create a readablestream. But as I am doing this as a cron job on a read-only file system (webtask.io) I'd want to achieve the same result without saving the file to disk temporarily. Shouldn't that be possible?
request(image.Url)
.pipe(
fs
.createWriteStream(image.Id)
.on('finish', () => {
client.assets
.upload('image', fs.createReadStream(image.Id))
.then(imageAsset => {
resolve(imageAsset)
})
})
)
Do you have any suggestions of how to achieve this without saving the file to disk? The upload client will take the following
client.asset.upload(type: 'file' | image', body: File | Blob | Buffer | NodeStream, options = {}): Promise<AssetDocument>
Thanks!
How about passing the buffer down to the upload function? Since as per your statement it'll accept a buffer.
As a side note... This will keep it in memory for the duration of the method execution, so if you call this numerous times you might run out of resources.
request.get(url, function (res) {
var data = [];
res.on('data', function(chunk) {
data.push(chunk);
}).on('end', function() {
var buffer = Buffer.concat(data);
// Pass the buffer
client.asset.upload(type: 'buffer', body: buffer);
});
});
I tried some various libraries and it turns out that node-fetch provides a way to return a buffer. So this code works:
fetch(image.Url)
.then(res => res.buffer())
.then(buffer => client.assets
.upload('image', buffer, {filename: image.Id}))
.then(imageAsset => {
resolve(imageAsset)
})
well I know it has been a few years since the question was originally asked, but I have encountered this problem now, and since I didn't find an answer with a comprehensive example I made one myself.
i'm assuming that the file path is a valid URL and that the end of it is the file name, I need to pass an apikey to this API endpoint, and a successful upload sends me back a response with a token.
I'm using node-fetch and form-data as dependencies.
const fetch = require('node-fetch');
const FormData = require('form-data');
const secretKey = 'secretKey';
const downloadAndUploadFile = async (filePath) => {
const fileName = new URL(filePath).pathname.split("/").pop();
const endpoint = `the-upload-endpoint-url`;
const formData = new FormData();
let jsonResponse = null;
try {
const download = await fetch(filePath);
const buffer = await download.buffer();
if (!buffer) {
console.log('file not found', filePath);
return null;
}
formData.append('file', buffer, fileName);
const response = await fetch(endpoint, {
method: 'POST', body: formData, headers: {
...formData.getHeaders(),
"Authorization": `Bearer ${secretKey}`,
},
});
jsonResponse = await response.json();
} catch (error) {
console.log('error on file upload', error);
}
return jsonResponse ? jsonResponse.token : null;
}