Develop Reference

Getting 400 error code when I run axios get request?

I write some code to getting info
const stock = await Stock.find({
exchange: exchange
});
// Here stock array length is 5300
stock.forEach(async (stockEl) => {
const EOD_API = process.env.EOD_HISTORICAL_API
const {data} = await axios.get(`https://eodhistoricaldata.com/api/fundamentals/${stockEl.code}?api_token=${EOD_API}&filter=General::Industry`);
console.log(data);
});
Here I place get request for every stock array element by forEach function. Then it give me error like image-
Click to see images
But When I place it outside of forEach function like this-
const EOD_API = process.env.EOD_HISTORICAL_API
const {data} = await axios.get(`https://eodhistoricaldata.com/api/fundamentals/${stockEl.code}?api_token=${EOD_API}&filter=General::Industry`);
console.log(data);
Then it gives no error. For Remembering Stock has 5300 element, that means axios run 5300 times.
Any solution or idea?

You need to make a few changes:
Replace forEach with for because forEach is not promise aware
Use try, catch => catch any errors
Use Promise.allSettled => it allows you to run all promisses together without waiting each other which in return will enhance your app performance. It returns an array with status ("fulfilled", "rejected")
const fetchSingleStockElement = async (stockEl) => {
try {
const EOD_API = process.env.EOD_HISTORICAL_API,
{ data } = await axios(
`https://eodhistoricaldata.com/api/fundamentals/${stockEl.code}?api_token=${EOD_API}&filter=General::Industry`
);
return data;
} catch (err) {
throw new Error(err);
}
};
const fetchAllStockData = async () => {
let promisesArray = [];
try {
//fetch stock array
const { data } = await Stock.find({
exchange: exchange
});
//fetch single stock
for (let i = 0; i < data.length; i++) {
promisesArray.push(fetchSingleStockElement(data[i].id));
}
const results = await Promise.allSettled(promisesArray);
console.log('results', results);
} catch (err) {
console.log('results error', err);
}
};
Here is a working example with fake API of 4466 entries:
const fetchSingleAirline = async (airlineId) => {
try {
const { data } = await axios(`https://api.instantwebtools.net/v1/airlines/${airlineId}`);
return data;
} catch (err) {
throw new Error(err);
}
};
const fetchAllAirlineData = async () => {
let promisesArray = [];
try {
const { data } = await axios('https://api.instantwebtools.net/v1/airlines');
for (let i = 0; i < data.length; i++) {
promisesArray.push(fetchSingleAirline(data[i].id));
}
const results = await Promise.allSettled(promisesArray);
console.log('results', results);
} catch (err) {
console.log('results error', err);
}
};

Doing await in forEach doesn't hold the process since forEach is not promise-aware. Try this instead:
(async () => {
for (let index = 0; index < stock.length; index++) {
const EOD_API = process.env.EOD_HISTORICAL_API
const {data} = await axios.get(`https://eodhistoricaldata.com/api/fundamentals/${stock[i].code}?api_token=${EOD_API}&filter=General::Industry`);
console.log(data);
}
})();
More information.

Run The Build Synchronously

I couldn't get my for loop to run synchronously.The order of registration that it returns for each domain changes. I guess whichever answers the fastest brings those records first. Where am I doing wrong?
const rrtypes= ["A","MX","CNAME","NS","TXT"];
var resData = [];
export const getAllRecords = async (req,res) => {
const {domain} = req.params;
for await(const rrtype of rrtypes){
dns.resolve (domain, rrtype, (err, records) => {
resData.push(rrtype+" "+records);
});
}
res.send(resData);
resData = [];
}

Notice that you can use the dns.promises.resolve function: https://nodejs.org/api/dns.html#dns_dnspromises_resolve_hostname_rrtype
so you would change your code to:
const rrtypes = ["A", "MX", "CNAME", "NS", "TXT"];
export const getAllRecords = async (req, res) => {
const { domain } = req.params;
// Notice that resData should be local to this function
let resData = [];
for (const rrtype of rrtypes) {
try {
const records = await dns.promises.resolve(domain, rrtype);
resData.push(rrtype + " " + records);
// IMO this would be better: resData.push({ type: rrtype, value: records });
} catch (err) {
// Log the error here
}
}
res.send(resData);
};

The problem lies in your use of await. await should be used when calling async functions. At the moment you are using it when instantiating variables from an array.
Thanks to a comment by slebetman I was able to see this dns.resolve does not return a promise but uses callbacks to manage the async calls. As he also suggested we can fix this by creating a promise to manage these callbacks.
const rrtypes= ["A","MX","CNAME","NS","TXT"];
var resData = [];
const dnsResolvePromise = (domain, rrtype) => {
return new Promise((resolve, reject) => {
dns.resolve(domain, rrtype, (err, records) => {
if(err) return reject(err);
resolve(rrtype+" "+records);
});
})
}
export const getAllRecords = async (req,res) => {
const {domain} = req.params;
for(const rrtype of rrtypes){
try{
const records = await dnsResolvePromise(domain, rrtype);
resData.push(records);
} catch (e){
// Handle error
}
}
res.send(resData);
resData = [];
}

Async/Await in for loop NodeJS Not blocking the loop execuation

I know that old school for loop works in the traditional way - that it waits for the await to finish getting results.
But in my use case, I need to read a file from local/s3 and process it line by line, and for each line I need to call an External API.
Generally I use await inside the loop because all are running inside a lambda and I don't want to use all memory for running it parallelly.
Here I am reading the file using a stream.on() method, and in order to use await inside that, I need to add async in read method, like so:
stream.on('data',async () =>{
while(data=stream.read()!==null){
console.log('line');
const requests = getRequests(); // sync code,no pblms
for(let i=0;i<requests.length;i++){
const result = await apiCall(request[i);
console.log('result from api')
const finalResult = await anotherapiCall(result.data);
}
}
});
This is working but order in which the lines are processed is not guaranteed. I need all in a sync manner. Any help?
Complete Code
async function processSOIFileLocal (options, params) {
console.log('Process SOI file');
const readStream = byline.createStream(fs.createReadStream(key));
readStream.setEncoding('utf8');
const pattern = /^UHL\s|^UTL\s/;
const regExp = new RegExp(pattern);
readStream.on('readable', () => {
let line;
while (null !== (line = readStream.read())) {
if (!regExp.test(line.toString())) {
totalRecordsCount++;
dataObject = soiParser(line);
const { id } = dataObject;
const XMLRequests = createLoSTRequestXML(
options,
{ mapping: event.mapping, row: dataObject }
);
console.log('Read line');
console.log(id);
try {
for (let i = 0;i < XMLRequests.length;i++) {
totalRequestsCount++;
console.log('Sending request');
const response = await sendLoSTRequest(
options,
{ data: XMLRequests[i],
url: LOST_URL }
);
console.log("got response");
const responseObj = await xml2js.
parseStringPromise(response.data);
if (Object.keys(responseObj).indexOf('errors') !== -1) {
fs.writeFileSync(`${ERR_DIR}/${generateKey()}-${id}.xml`, response.data);
failedRequestsCount++;
} else {
successRequestsCount++;
console.log('Response from the Lost Server');
console.log(response[i].data);
}
}
} catch (err) {
console.log(err);
}
}
}
})
.on('end', () => {
console.log('file processed');
console.log(`
************************************************
Total Records Processed:${totalRecordsCount}
Total Requests Sent: ${totalRequestsCount}
Success Requests: ${successRequestsCount}
Failed Requests: ${failedRequestsCount}
************************************************
`);
});
}
async function sendLoSTRequest (options, params) {
const { axios } = options;
const { url, data } = params;
if (url) {
return axios.post(url, data);
// eslint-disable-next-line no-else-return
} else {
console.log('URL is not found');
return null;
}
}
Code needs to flow like so:
read a line in a sync way
process the line and transform the line into an array of two members
for every member call API and do stuff
once line is complete, look for another line, all done in order
UPDATE: Got a workaround..but it fires stream.end() without waiting stream to finish read
async function processSOIFileLocal (options, params) {
console.log('Process SOI file');
const { ERR_DIR, fs, xml2js, LOST_URL, byline, event } = options;
const { key } = params;
const responseObject = {};
let totalRecordsCount = 0;
let totalRequestsCount = 0;
let failedRequestsCount = 0;
let successRequestsCount = 0;
let dataObject = {};
const queue = (() => {
let q = Promise.resolve();
return fn => (q = q.then(fn));
})();
const readStream = byline.createStream(fs.createReadStream(key));
readStream.setEncoding('utf8');
const pattern = /^UHL\s|^UTL\s/;
const regExp = new RegExp(pattern);
readStream.on('readable', () => {
let line;
while (null !== (line = readStream.read())) {
if (!regExp.test(line.toString())) {
totalRecordsCount++;
dataObject = soiParser(line);
const { id } = dataObject;
const XMLRequests = createLoSTRequestXML(
options,
{ mapping: event.mapping, row: dataObject }
);
// eslint-disable-next-line no-loop-func
queue(async () => {
try {
for (let i = 0;i < XMLRequests.length;i++) {
console.log('Sending request');
console.log(id);
totalRequestsCount++;
const response = await sendLoSTRequest(
options,
{ data: XMLRequests[i],
url: LOST_URL }
);
console.log('got response');
const responseObj = await xml2js.
parseStringPromise(response.data);
if (Object.keys(responseObj).indexOf('errors') !== -1) {
// console.log('Response have the error:');
// await handleError(options, { err: responseObj, id });
failedRequestsCount++;
fs.writeFileSync(`${ERR_DIR}/${generateKey()}-${id}.xml`, response.data);
} else {
console.log('Response from the Lost Server');
console.log(response[i].data);
successRequestsCount++;
}
}
} catch (err) {
console.log(err);
}
});
}
}
})
.on('end', () => {
console.log('file processed');
console.log(`
************************************************
Total Records Processed:${totalRecordsCount}
Total Requests Sent: ${totalRequestsCount}
Success Requests: ${successRequestsCount}
Failed Requests: ${failedRequestsCount}
************************************************
`);
Object.assign(responseObject, {
failedRequestsCount,
successRequestsCount,
totalRecordsCount,
totalRequestsCount
});
});
}
Thank You

The sample code at the top of your question could be rewritten like
const queue = (() => {
let q = Promise.resolve();
return (fn) => (q = q.then(fn));
})();
stream.on('data', async() => {
while (data = stream.read() !== null) {
console.log('line');
const requests = getRequests(); // sync code,no pblms
queue(async () => {
for (let i = 0; i < requests.length; i++) {
const result = await apiCall(request[i]);
console.log('result from api');
const finalResult = await anotherapiCall(result.data);
}
});
}
});
Hopefully that will be useful for the complete code

If anyone want a solution for synchronisely process the file, ie, linebyline read and execute some Async call, it's recommended to use inbuilt stream transform. There we can create a transform function and return a callback when finishes.
That's will help of any one face this issues.
Through2 is a small npm library that also can be used for the same.

variable initialisation node js

here's my problem in this function I'm trying to return an array of objects. When i do a console.log in the forEach loop as you can see the array is filled but when I try to print it juste before my return statement it's empty.
and here's my full js file :
const Discord = require("discord.js");
const Maths = require("../Utils/Maths");
const Stop = require("../Commands/stop");
const EventMitter = require("events");
const emitter = new EventMitter();
const fs = require("fs");
module.exports.play = async(client, channel, players) => {
players.push(client.guilds.cache.first().members.cache.array().filter(mem => mem.user.username === "user1").map(mem => mem.user)[0]);
players.push(client.guilds.cache.first().members.cache.array().filter(mem => mem.user.username === "users2").map(mem => mem.user)[0]);
try {
await beginMessage(players, channel);
let playersCards = await beginGame(client, players);
console.log(playersCards);
emitter.on("stop", async () => {
await channel.delete();
console.log("jeu fini");
});
}
catch(e) {
console.error(e);
}
};
async function beginMessage(players, channel) {
let message = "que le jeu commence ";
players.forEach(player => {
message += `${player.toString()} `;
});
await channel.send(message);
}
async function beginGame(client, players) {
let playersCards = [];
fs.readFile("Utils/cartes.json", 'utf8', async (err, data) => {
if(err) console.log(err);
let cartes = JSON.parse(data);
cartes = cartes.cartes;
players.forEach(async player => {
let playerCards = await distributeCards(player, cartes);
playersCards.push(playerCards);
//console.log(playersCards);
});
console.log(playersCards);
return playersCards;
});
}
async function distributeCards(player, cartes) {
let playerCards = [];
for(let i = 0; i < 4; i++) {
let carte = cartes[Maths.getRandomNumber(0,12)];
carte.count--;
playerCards.push(carte.name);
}
let dmChannel = await player.createDM();
await dmChannel.send(playerCards);
return playerCards;
}
module.exports.finishGame = function (client) {
emitter.emit("stop");
};
Must be a initialisation mistake or something like that I've been searching in my code but can't find it.

EDIT 3:
Since fs.readFile is not async we need to convert it to a promised based function something like below.
function readFile(path) {
return new Promise((resolve, reject) => {
fs.readFile(path, 'utf8', function (err, data) {
if (err) {
reject(err);
}
resolve(data);
});
});
}
I have updated the codesandbox snippet. Please check and see it that solves your issue.
https://codesandbox.io/s/nodejs-async-foreach-promise-v16w6
EDIT 2:
You need to implement custom forEach that will execute and return values as async. Something like this:
async function asyncForEach(array, callback) {
for (let index = 0; index < array.length; index++) {
await callback(array[index], index, array);
}
}
I have written a codesandbox snippet that might help you.
https://codesandbox.io/s/nodejs-async-foreach-promise-v16w6
EDIT:
Since you are using promise inside the forEach loop, the values are pushed
only after the promise is resolved. Since forEach is completes even before the values are pushed to playersCards you will not be able to see any values.
You can do something like this. I have updated the code. Haven't tested the code, but can help if there is issue.
async fs.readFile("Utils/cartes.json", 'utf8', async (err, data) => {
if(err) console.log(err);
let cartes = JSON.parse(data);
cartes = cartes.cartes;
await players.forEach(async player => {
let playerCardsPromiseValue = await distributeCards(player, cartes);
playersCards.push(playerCardsPromiseValue);
//console.log(playersCards);
});
console.log(playersCards);
return playersCards;
});

How to run promise based function serially

I am trying to run the node js Lighthouse function serially (one at a time) with an array of URLs. My problem is that whenever I loop through the array, Lighthouse runs all the URLs at once, which I imagine is problematic if you have a very large array of URLs.
The code:
for(let url of urls) {
function launchChromeAndRunLighthouse(url, opts, config = null) {
return chromeLauncher.launch({chromeFlags: opts.chromeFlags}).then(chrome => {
opts.port = chrome.port;
return lighthouse(url, opts, config).then(results => {
return chrome.kill().then(() => results.lhr)
});
});
}
}
launchChromeAndRunLighthouse('https://example.com', opts).then(results => {
// Use results!
});
Please help! And thank you for your time!

Your answer is correct but it can be improved. Since you have access to async and await, you should fully utilize it to make your code cleaner:
async function launchChromeAndRunLighthouse (url, opts, config = null) {
const chrome = await chromeLauncher.launch({chromeFlags: opts.chromeFlags});
opts.port = chrome.port;
const { lhr } = await lighthouse(url, opts, config);
await chrome.kill();
return lhr;
}
async function launchAudit (urls) {
for (const url of urls) {
const results = await launchChromeAndRunLighthouse(url, opts);
// Use results!
};
}
launchAudit(urls);

I believe I figured it out. What I did is below. Please continue to send feedback if you think this is wrong.
function launchChromeAndRunLighthouse(url, opts, config = null) {
return chromeLauncher.launch({chromeFlags: opts.chromeFlags}).then(chrome => {
opts.port = chrome.port;
return lighthouse(url, opts, config).then(results => {
return chrome.kill().then(() => results.lhr)
});
});
};
async function launchAudit(urls) {
for (let url of urls) {
await launchChromeAndRunLighthouse(url, opts).then(results => {
// Use results!
});
};
};
launchAudit(urls);

A variation on Patric Roberts answer (which should be the accepted answer).
I was wondering if it was necessary to kill chrome every iteration.
const lighthouse = require('lighthouse');
const chromeLauncher = require('chrome-launcher');
function launchChromeAndRunLighthouse(sites, opts, config = null) {
return chromeLauncher.launch({chromeFlags: opts.chromeFlags}).then(chrome => {
opts.port = chrome.port;
const siteResults = [];
return new Promise((resolve, reject) => {
// batch async functions.
// C/O https://stackoverflow.com/questions/43082934/how-to-execute-promises-sequentially-passing-the-parameters-from-an-array
const runBatch = async (iterable, action) => {
for (const x of iterable) {
await action(x)
}
}
// func to run lighthouse
const doLightHouse = (site) => new Promise((resolve, reject) => {
lighthouse(site, opts, config).then(results => {
siteResults.push(results.lhr);
resolve();
});
});
// go go go
runBatch(sites, doLightHouse).then(d => {
chrome.kill().then((result) => {
resolve(siteResults)
})
});
});
});
}
const opts = {
chromeFlags: ['--show-paint-rects'],
onlyCategories: ['performance']
};
const sites = ['https://www.example.com', 'https://www.test.com']
launchChromeAndRunLighthouse(sites, opts).then(results => {
// Use results!
console.log(results);
});

Just to execute your code as test, we'll use async/await and IIFE
Then, will create function which will put all our request to array of non resolved promises, so we could use it with Promise.all()
You need to rewrite code in something like this:
(async() => {
const promisesToExecute = [];
const launchChromeAndRunLighthouse = async (url, opts, config = null) => {
const chrome = await return chromeLauncher.launch({chromeFlags: opts.chromeFlags});
opts.port = chrome.port;
promisesToExecute.push(lighthouse(url, opts, config));
}
const results = await Promise.all(promisesToExecute);
for(const result of results) {
const resolvedResult = await result.kill();
// here you can access your results.lhr
console.log(resolvedResult.lhr);
}
})()
Please note, this code wasn't tested, so there might be problems with kill() on result. But, the main goal is to answer your question and explain how to execute promises.
Also, if you don't want to execute all promises at the same time, you could use Promise.waterfall with some npm package, like this