Develop Reference

Node.js save audio file from mediaserver [duplicate]

i want download a pdf file with axios and save on disk (server side) with fs.writeFile, i have tried:
axios.get('https://xxx/my.pdf', {responseType: 'blob'}).then(response => {
fs.writeFile('/temp/my.pdf', response.data, (err) => {
if (err) throw err;
console.log('The file has been saved!');
});
});
the file is saved but the content is broken...
how do I correctly save the file?

Actually, I believe the previously accepted answer has some flaws, as it will not handle the writestream properly, so if you call "then()" after Axios has given you the response, you will end up having a partially downloaded file.
This is a more appropriate solution when downloading slightly larger files:
export async function downloadFile(fileUrl: string, outputLocationPath: string) {
const writer = createWriteStream(outputLocationPath);
return Axios({
method: 'get',
url: fileUrl,
responseType: 'stream',
}).then(response => {
//ensure that the user can call `then()` only when the file has
//been downloaded entirely.
return new Promise((resolve, reject) => {
response.data.pipe(writer);
let error = null;
writer.on('error', err => {
error = err;
writer.close();
reject(err);
});
writer.on('close', () => {
if (!error) {
resolve(true);
}
//no need to call the reject here, as it will have been called in the
//'error' stream;
});
});
});
}
This way, you can call downloadFile(), call then() on the returned promise, and making sure that the downloaded file will have completed processing.
Or, if you use a more modern version of NodeJS, you can try this instead:
import * as stream from 'stream';
import { promisify } from 'util';
const finished = promisify(stream.finished);
export async function downloadFile(fileUrl: string, outputLocationPath: string): Promise<any> {
const writer = createWriteStream(outputLocationPath);
return Axios({
method: 'get',
url: fileUrl,
responseType: 'stream',
}).then(response => {
response.data.pipe(writer);
return finished(writer); //this is a Promise
});
}

You can simply use response.data.pipe and fs.createWriteStream to pipe response to file
axios({
method: "get",
url: "https://xxx/my.pdf",
responseType: "stream"
}).then(function (response) {
response.data.pipe(fs.createWriteStream("/temp/my.pdf"));
});

The problem with broken file is because of backpressuring in node streams. You may find this link useful to read: https://nodejs.org/es/docs/guides/backpressuring-in-streams/
I'm not really a fan of using Promise base declarative objects in JS codes as I feel it pollutes the actual core logic & makes the code hard to read. On top of it, you have to provision event handlers & listeners to make sure the code is completed.
A more cleaner approach on the same logic which the accepted answer proposes is given below. It uses the concepts of stream pipelines.
const util = require('util');
const stream = require('stream');
const pipeline = util.promisify(stream.pipeline);
const downloadFile = async () => {
try {
const request = await axios.get('https://xxx/my.pdf', {
responseType: 'stream',
});
await pipeline(request.data, fs.createWriteStream('/temp/my.pdf'));
console.log('download pdf pipeline successful');
} catch (error) {
console.error('download pdf pipeline failed', error);
}
}
exports.downloadFile = downloadFile
I hope you find this useful.

// This works perfectly well!
const axios = require('axios');
axios.get('http://www.sclance.com/pngs/png-file-download/png_file_download_1057991.png', {responseType: "stream"} )
.then(response => {
// Saving file to working directory
response.data.pipe(fs.createWriteStream("todays_picture.png"));
})
.catch(error => {
console.log(error);
});

The following code taken from https://gist.github.com/senthilmpro/072f5e69bdef4baffc8442c7e696f4eb?permalink_comment_id=3620639#gistcomment-3620639 worked for me
const res = await axios.get(url, { responseType: 'arraybuffer' });
fs.writeFileSync(downloadDestination, res.data);

node fileSystem writeFile encodes data by default to UTF8. which could be a problem in your case.
Try setting your encoding to null and skip encoding the received data:
fs.writeFile('/temp/my.pdf', response.data, {encoding: null}, (err) => {...}
you can also decalre encoding as a string (instead of options object) if you only declare encoding and no other options. string will be handled as encoding value. as such:
fs.writeFile('/temp/my.pdf', response.data, 'null', (err) => {...}
more read in fileSystem API write_file

I have tried, and I'm sure that using response.data.pipe and fs.createWriteStream can work.
Besides, I want to add my situation and solution
Situation:
using koa to develop a node.js server
using axios to get a pdf via url
using pdf-parse to parse the pdf
extract some information of pdf and return it as json to browser
Solution:
const Koa = require('koa');
const app = new Koa();
const axios = require('axios')
const fs = require("fs")
const pdf = require('pdf-parse');
const utils = require('./utils')
app.listen(process.env.PORT || 3000)
app.use(async (ctx, next) => {
let url = 'https://path/name.pdf'
let resp = await axios({
url: encodeURI(url),
responseType: 'arraybuffer'
})
let data = await pdf(resp.data)
ctx.body = {
phone: utils.getPhone(data.text),
email: utils.getEmail(data.text),
}
})
In this solution, it doesn't need to write file and read file, it's more efficient.

This is what worked for me and it also creates a temporary file for the image file in case the output file path is not specified:
const fs = require('fs')
const axios = require('axios').default
const tmp = require('tmp');
const downloadFile = async (fileUrl, outputLocationPath) => {
if(!outputLocationPath) {
outputLocationPath = tmp.fileSync({ mode: 0o644, prefix: 'kuzzle-listener-', postfix: '.jpg' });
}
let path = typeof outputLocationPath === 'object' ? outputLocationPath.name : outputLocationPath
const writer = fs.createWriteStream(path)
const response = await axios.get(fileUrl, { responseType: 'arraybuffer' })
return new Promise((resolve, reject) => {
if(response.data instanceof Buffer) {
writer.write(response.data)
resolve(outputLocationPath.name)
} else {
response.data.pipe(writer)
let error = null
writer.on('error', err => {
error = err
writer.close()
reject(err)
})
writer.on('close', () => {
if (!error) {
resolve(outputLocationPath.name)
}
})
}
})
}
Here is a very simple Jest test:
it('when downloadFile should downloaded', () => {
downloadFile('https://i.ytimg.com/vi/HhpbzPMCKDc/hq720.jpg').then((file) => {
console.log('file', file)
expect(file).toBeTruthy()
expect(file.length).toBeGreaterThan(10)
})
})

Lorenzo's response is probably the best answer as it's using the axios built-in.
Here's a simple way to do it if you just want the buffer:
const downloadFile = url => axios({ url, responseType: 'stream' })
.then(({ data }) => {
const buff = []
data.on('data', chunk => buff.push(chunk))
return new Promise((resolve, reject) => {
data.on('error', reject)
data.on('close', () => resolve(Buffer.concat(buff)))
})
})
// better
const downloadFile = url => axios({ url, responseType: 'arraybuffer' }).then(res => res.data)
const res = await downloadFile(url)
fs.writeFileSync(downloadDestination, res)
I'd still probably use the 'arraybuffer' responseType

There is a much simpler way that can be accomplished in a couple of lines:
const fileResponse = await axios({
url: fileUrl,
method: "GET",
responseType: "stream",
});
// Write file to disk (here I use fs.promise but you can use writeFileSync it's equal
await fsPromises.writeFile(filePath, fileResponse.data);
Axios has internal capacity of handling streams and you don't need to necessarily meddle with low-level Node APIs for that.
Check out https://axios-http.com/docs/req_config (find the responseTypepart in the docs for all the types you can use).

If you just want the file use this
const media_data =await axios({url: url, method: "get", responseType: "arraybuffer"})
writeFile("./image.jpg", Buffer.from(media_data.data), {encoding: "binary"}, console.log)

import download from "downloadjs";
export const downloadFile = async (fileName) => {
axios({
method: "get",
url: `/api/v1/users/resume/${fileName}`,
responseType: "blob",
}).then(function (response) {
download(response.data, fileName);
});
};
it's work fine to me

This is my example code run with node js
There is a synctax error
should be writeFile not WriteFile
const axios = require('axios');
const fs = require('fs');
axios.get('http://www.africau.edu/images/default/sample.pdf', {responseType: 'blob'}).then(response => {
fs.writeFile('./my.pdf', response.data, (err) => {
if (err) throw err;
console.log('The file has been saved!');
});
});
After the file is saved it might look like in a text editor, but the file was saved properly
%PDF-1.3
%����
1 0 obj
<<
/Type /Catalog
/Outlines 2 0 R
/Pages 3 0 R
>>
endobj
2 0 obj
<<
/Type /Outlines
/Count 0
>>
endobj
3 0 obj
<<
/Type /Pages
/Count 2
/Kids [ 4 0 R 6 0 R ]
>>
endobj

React App URL and Backend URL are getting concatenated when fetching data using axios

In my full stack application, I got the app folder and a server and client folder inside.
When going to X URL a useEffect is triggered, and it should fetch data from localhost:4000/pets/cats, but instead I get an error:
http://localhost:3000/localhost:4000/pets/cats 404 (Not Found)
For some reason it's trying to look for this URL: http://localhost:3000/localhost:4000/pets/cats (that doesn't really exist). How to fix it?
Here's the get method
The reducer
The useEffect where I should be getting the info

You need to set a proper base URL for axios:
An example using full URL:
useEffect(() => {
(async () => {
try {
const url = "http://localhost:4000/pets/cats"; // Full URL
await axios.get(url);
} catch (err) {
console.log("Some error: ", err);
}
})();
}, []);
An example using base URL:
useEffect(() => {
(async () => {
try {
const url = "/pets/cats";
await axios.get(url, {
baseURL: "http://localhost:4000", // Base URL
});
} catch (err) {
console.log("Some error: ", err);
}
})();
}, []);

you try to call api twice so api is not found or 404, try this localhost:4000/pets/cats

Phonegap: Is there a way to get JSON data from an external url?

Actually am kinda disappointed as I tried many things and checked out many articles but non worked out for me.
function demo() {
console.log("Booooooooooooommmmmmmmmmm");
tokenV = document.getElementById("tokenString").value;
var urlF = "https://***********.com/connect/api.php?action=2&token="+tokenV;
const myHeaders = new Headers();
const myRequest = new Request(urlF, {
method: 'GET',
headers: myHeaders,
mode: 'cors',
cache: 'default',
});
fetch(myRequest)
.then(response => response.json())
.then(data => console.log(data));
}
I have already whitlist the domain inside my config file, am using phonegap CL latest version. I'm trying to connect to an api which will out put json.encode data if token were right.
Error output:
(index):50 Fetch failed loading: GET https://*******.com/connect/api.php.............
Another way I tried using cordova fetch plugin still failed:
function demo() {
console.log("Booooooooooooommmmmmmmmmm");
tokenV = document.getElementById("tokenString").value;
var urlF = "https://*********.com/api.php?action=2&token="+tokenV;
console.log("nowww1");
cordovaFetch(urlF, {
method : 'GET',
headers: {
'User-Agent': 'CordovaFetch 1.0.0'
},
})
.then(function(response) {
return response.json();
}).then(function(json) {
console.log('parsed json', json);
}).catch(function(ex) {
console.log('parsing failed', ex);
});
}
Error out put:
Error: exec proxy not found for :: FetchPlugin :: fetch (index):118 parsing failed TypeError: Network request failed
I can change the out put as I want but show me away to get the data from an external server???
Thank you

node.js axios download file stream and writeFile

i want download a pdf file with axios and save on disk (server side) with fs.writeFile, i have tried:
axios.get('https://xxx/my.pdf', {responseType: 'blob'}).then(response => {
fs.writeFile('/temp/my.pdf', response.data, (err) => {
if (err) throw err;
console.log('The file has been saved!');
});
});
the file is saved but the content is broken...
how do I correctly save the file?

Actually, I believe the previously accepted answer has some flaws, as it will not handle the writestream properly, so if you call "then()" after Axios has given you the response, you will end up having a partially downloaded file.
This is a more appropriate solution when downloading slightly larger files:
export async function downloadFile(fileUrl: string, outputLocationPath: string) {
const writer = createWriteStream(outputLocationPath);
return Axios({
method: 'get',
url: fileUrl,
responseType: 'stream',
}).then(response => {
//ensure that the user can call `then()` only when the file has
//been downloaded entirely.
return new Promise((resolve, reject) => {
response.data.pipe(writer);
let error = null;
writer.on('error', err => {
error = err;
writer.close();
reject(err);
});
writer.on('close', () => {
if (!error) {
resolve(true);
}
//no need to call the reject here, as it will have been called in the
//'error' stream;
});
});
});
}
This way, you can call downloadFile(), call then() on the returned promise, and making sure that the downloaded file will have completed processing.
Or, if you use a more modern version of NodeJS, you can try this instead:
import * as stream from 'stream';
import { promisify } from 'util';
const finished = promisify(stream.finished);
export async function downloadFile(fileUrl: string, outputLocationPath: string): Promise<any> {
const writer = createWriteStream(outputLocationPath);
return Axios({
method: 'get',
url: fileUrl,
responseType: 'stream',
}).then(response => {
response.data.pipe(writer);
return finished(writer); //this is a Promise
});
}

You can simply use response.data.pipe and fs.createWriteStream to pipe response to file
axios({
method: "get",
url: "https://xxx/my.pdf",
responseType: "stream"
}).then(function (response) {
response.data.pipe(fs.createWriteStream("/temp/my.pdf"));
});

The problem with broken file is because of backpressuring in node streams. You may find this link useful to read: https://nodejs.org/es/docs/guides/backpressuring-in-streams/
I'm not really a fan of using Promise base declarative objects in JS codes as I feel it pollutes the actual core logic & makes the code hard to read. On top of it, you have to provision event handlers & listeners to make sure the code is completed.
A more cleaner approach on the same logic which the accepted answer proposes is given below. It uses the concepts of stream pipelines.
const util = require('util');
const stream = require('stream');
const pipeline = util.promisify(stream.pipeline);
const downloadFile = async () => {
try {
const request = await axios.get('https://xxx/my.pdf', {
responseType: 'stream',
});
await pipeline(request.data, fs.createWriteStream('/temp/my.pdf'));
console.log('download pdf pipeline successful');
} catch (error) {
console.error('download pdf pipeline failed', error);
}
}
exports.downloadFile = downloadFile
I hope you find this useful.

// This works perfectly well!
const axios = require('axios');
axios.get('http://www.sclance.com/pngs/png-file-download/png_file_download_1057991.png', {responseType: "stream"} )
.then(response => {
// Saving file to working directory
response.data.pipe(fs.createWriteStream("todays_picture.png"));
})
.catch(error => {
console.log(error);
});

The following code taken from https://gist.github.com/senthilmpro/072f5e69bdef4baffc8442c7e696f4eb?permalink_comment_id=3620639#gistcomment-3620639 worked for me
const res = await axios.get(url, { responseType: 'arraybuffer' });
fs.writeFileSync(downloadDestination, res.data);

node fileSystem writeFile encodes data by default to UTF8. which could be a problem in your case.
Try setting your encoding to null and skip encoding the received data:
fs.writeFile('/temp/my.pdf', response.data, {encoding: null}, (err) => {...}
you can also decalre encoding as a string (instead of options object) if you only declare encoding and no other options. string will be handled as encoding value. as such:
fs.writeFile('/temp/my.pdf', response.data, 'null', (err) => {...}
more read in fileSystem API write_file

I have tried, and I'm sure that using response.data.pipe and fs.createWriteStream can work.
Besides, I want to add my situation and solution
Situation:
using koa to develop a node.js server
using axios to get a pdf via url
using pdf-parse to parse the pdf
extract some information of pdf and return it as json to browser
Solution:
const Koa = require('koa');
const app = new Koa();
const axios = require('axios')
const fs = require("fs")
const pdf = require('pdf-parse');
const utils = require('./utils')
app.listen(process.env.PORT || 3000)
app.use(async (ctx, next) => {
let url = 'https://path/name.pdf'
let resp = await axios({
url: encodeURI(url),
responseType: 'arraybuffer'
})
let data = await pdf(resp.data)
ctx.body = {
phone: utils.getPhone(data.text),
email: utils.getEmail(data.text),
}
})
In this solution, it doesn't need to write file and read file, it's more efficient.

This is what worked for me and it also creates a temporary file for the image file in case the output file path is not specified:
const fs = require('fs')
const axios = require('axios').default
const tmp = require('tmp');
const downloadFile = async (fileUrl, outputLocationPath) => {
if(!outputLocationPath) {
outputLocationPath = tmp.fileSync({ mode: 0o644, prefix: 'kuzzle-listener-', postfix: '.jpg' });
}
let path = typeof outputLocationPath === 'object' ? outputLocationPath.name : outputLocationPath
const writer = fs.createWriteStream(path)
const response = await axios.get(fileUrl, { responseType: 'arraybuffer' })
return new Promise((resolve, reject) => {
if(response.data instanceof Buffer) {
writer.write(response.data)
resolve(outputLocationPath.name)
} else {
response.data.pipe(writer)
let error = null
writer.on('error', err => {
error = err
writer.close()
reject(err)
})
writer.on('close', () => {
if (!error) {
resolve(outputLocationPath.name)
}
})
}
})
}
Here is a very simple Jest test:
it('when downloadFile should downloaded', () => {
downloadFile('https://i.ytimg.com/vi/HhpbzPMCKDc/hq720.jpg').then((file) => {
console.log('file', file)
expect(file).toBeTruthy()
expect(file.length).toBeGreaterThan(10)
})
})

Lorenzo's response is probably the best answer as it's using the axios built-in.
Here's a simple way to do it if you just want the buffer:
const downloadFile = url => axios({ url, responseType: 'stream' })
.then(({ data }) => {
const buff = []
data.on('data', chunk => buff.push(chunk))
return new Promise((resolve, reject) => {
data.on('error', reject)
data.on('close', () => resolve(Buffer.concat(buff)))
})
})
// better
const downloadFile = url => axios({ url, responseType: 'arraybuffer' }).then(res => res.data)
const res = await downloadFile(url)
fs.writeFileSync(downloadDestination, res)
I'd still probably use the 'arraybuffer' responseType

There is a much simpler way that can be accomplished in a couple of lines:
const fileResponse = await axios({
url: fileUrl,
method: "GET",
responseType: "stream",
});
// Write file to disk (here I use fs.promise but you can use writeFileSync it's equal
await fsPromises.writeFile(filePath, fileResponse.data);
Axios has internal capacity of handling streams and you don't need to necessarily meddle with low-level Node APIs for that.
Check out https://axios-http.com/docs/req_config (find the responseTypepart in the docs for all the types you can use).

If you just want the file use this
const media_data =await axios({url: url, method: "get", responseType: "arraybuffer"})
writeFile("./image.jpg", Buffer.from(media_data.data), {encoding: "binary"}, console.log)

import download from "downloadjs";
export const downloadFile = async (fileName) => {
axios({
method: "get",
url: `/api/v1/users/resume/${fileName}`,
responseType: "blob",
}).then(function (response) {
download(response.data, fileName);
});
};
it's work fine to me

This is my example code run with node js
There is a synctax error
should be writeFile not WriteFile
const axios = require('axios');
const fs = require('fs');
axios.get('http://www.africau.edu/images/default/sample.pdf', {responseType: 'blob'}).then(response => {
fs.writeFile('./my.pdf', response.data, (err) => {
if (err) throw err;
console.log('The file has been saved!');
});
});
After the file is saved it might look like in a text editor, but the file was saved properly
%PDF-1.3
%����
1 0 obj
<<
/Type /Catalog
/Outlines 2 0 R
/Pages 3 0 R
>>
endobj
2 0 obj
<<
/Type /Outlines
/Count 0
>>
endobj
3 0 obj
<<
/Type /Pages
/Count 2
/Kids [ 4 0 R 6 0 R ]
>>
endobj

node.js timeout restarts api call

Platform:
I have an api in sails.js and a frontend in react. The calls between front and back end are being made with fetch api.
More information:
In the course of some api endpoints I have to execute an external file, at this point I am using the execfile() function of node.js, and I have to wait for it to be executed to respond to the frontend.
What is the problem?
If the file is executed in a short time, for example less than 1 minute everything runs well and the behavior occurs as expected on both sides, but if (in this case) the file takes more than 1 minute to execute, there is something to trigger a second call to api (I do not know where this is being called, but I tested the same endpoint with postman and I did not have this problem so I suspect the react / fetch-api) and the api call with the same data from the first call is redone. This causes the file to run twice.
Something that is even stranger is that if you have the DevTools Network inspector turned on this second call does not appear, but nothing in the documentation of sailjs points to this behavior.
Example of an endpoint in sails.js:
/**
* FooController
*/
const execFile = require("child_process").execFile;
module.exports = {
foo: async (req, res) => {
let result = await module.exports._executeScript(req.body).catch(() => {
res.status(500).json({ error: "something has occurred" });
});
res.json(result);
},
_executeScript: body => {
return new Promise((resolve, reject) => {
let args = [process.cwd() + "/scripts/" + "myExternalFile.js", body];
let elem = await module.exports
._execScript(args)
.catch(err => reject(err));
resolve(elem);
});
},
_execScript: args => {
return new Promise((resolve, reject) => {
try {
execFile("node", args, { timeout: 150000 }, (error, stdout, stderr) => {
if (error || (stderr != null && stderr !== "")) {
console.error(error);
} else {
console.log(stdout);
}
let output = { stdout: stdout, stderr: stderr };
resolve(output);
});
} catch (err) {
reject(err);
}
});
}
};
Example of component react with fetch call:
import React from "react";
import { notification } from "antd";
import "./../App.css";
import Oauth from "./../helper/Oauth";
import Config from "./../config.json";
class App extends React.Component {
constructor(props) {
super(props);
this.state = {
syncInAction: false,
data: null
};
}
componentDidMount() {
this.handleSync();
}
async handleSync() {
let response = await fetch(Config.apiLink + "/foo/foo", {
method: "POST",
mode: "cors",
headers: {
Authorization: Oauth.isLoggedIn(),
Accept: "application/json",
"Content-Type": "application/json"
},
body: JSON.stringify(this.state.myData)
}).catch(err => {
notification.error({
key: "catch-ApiFail",
message: "Erro"
});
});
let json = await response.json();
this.setState({
syncInAction: false,
data: json
});
}
render() {
return <div>{this.state.data}</div>;
}
}
export default App;
What is my expected goal / behavior:
It does not matter if the call takes 1 minute or 10 hours, the file can only be called once and when it finishes, then yes, it can return to the frontend.
Note that the examples do not have the original code and have not been tested. Is a simplified version of the code to explain the behavior

I ended up solving the problem, apparently nodejs has a default timing of 2 minutes on the server, and can be rewritten to miss this timout or increase it.
This was just adding a line of code at the beginning of the foo() endpoint and the problem was solved.
The behavior of redoing the call is that it is not documented, and it is strange not to have this behavior when using the postman, but here is the solution for whoever needs it.
Final result:
foo: async (req, res) => {
req.setTimeout(0);
let result = await module.exports._executeScript(req.body).catch(() => {
res.status(500).json({ error: "something has occurred" });
});
res.json(result);
};