Develop Reference

Node fetch loop too slow

I have an API js file which I call with a POST method, passing in an array of objects which each contains a site url (about 26 objects or urls) as the body, and with the code below I loop through this array (sites) , check if each object url returns a json by adding to the url the "/items.json" , if so push the json content into another final array siteLists which I send back as response.
The problem is for just 26 urls, this API call takes more than 5 seconds to complete, am I doing it the wrong way or is it just the way fetch works in Node.js?
const sites content looks like:
[{label: "JonLabel", name: "Jon", url: "jonurl.com"},{...},{...}]
code is:
export default async (req, res) => {
if (req.method === 'POST') {
const body = JSON.parse(req.body)
const sites = body.list // this content shown above
var siteLists = []
if (sites?.length > 0){
var b=0, idd=0
while (b < sites.length){
let url = sites?.[b]?.url
if (url){
let jurl = `${url}/items.json`
try {
let fUrl = await fetch(jurl)
let siteData = await fUrl.json()
if (siteData){
let items = []
let label = sites?.[b]?.label || ""
let name = sites?.[b]?.name || ""
let base = siteData?.items
if(base){
var c = 0
while (c < base.length){
let img = base[c].images[0].url
let titl = base[c].title
let obj = {
url: url,
img: img,
title: titl
}
items.push(obj)
c++
}
let object = {
id: idd,
name: name,
label: label,
items: items
}
siteLists.push(object)
idd++
}
}
}catch(err){
//console.log(err)
}
}
b++
}
res.send({ sites: siteLists })
}
res.end()
}
EDIT: (solution?)
So it seems the code with promises as suggested below and marked as the solution works in the sense that is faster, the funny thing tho is it still takes more than 5 secs to load and still throws a Failed to load resource: the server responded with a status of 504 (Gateway Time-out) error, since Vercel, where the app is hosted passed to a max timeout of 5 secs for serverless functions, therefore never loading the content in the response. Locally, where I got no timeout limits is visibly faster to load, but it surprises me that such a query takes so long to complete where it should be a matter of ms.

The biggest problem I see here is that you appear to be awaiting for one fetch to complete before you loop through to start the next fetch request, effectively running them serially. If you rewrote your script to run all of the simultaneously in parallel, you could push each request sequentially into a Promise.all and then process the results when they return.
Think of it like this-- if each request took a second to complete, and you have 26 requests, and you wait for one to complete before starting the next, it will take 26 seconds altogether. However, if you run them each all together, if they still each take only one second to complete the whole thing altogether will take just one second.
An example in psuedocode--
You want to change this:
const urls = ['url1', 'url2', 'url3'];
for (let url of urls) {
const result = await fetch(url);
process(result)
}
...into this:
const urls = ['url1', 'url2', 'url3'];
const requests = [];
for (let url of urls) {
requests.push(fetch(url));
}
Promise.all(requests)
.then(
(results) => results.forEach(
(result) => process(result)
)
);

While await is a great sugar, sometimes it's better to stick with then
export default async (req, res) => {
if (req.method === 'POST') {
const body = JSON.parse(req.body)
const sites = body.list // this content shown above
const siteListsPromises = []
if (sites?.length > 0){
var b=0
while (b < sites.length){
let url = sites?.[b]?.url
if (url) {
let jurl = `${url}/items.json`
// #1
const promise = fetch(jurl)
// #2
.then(async (fUrl) => {
let siteData = await fUrl.json()
if (siteData){
...
return {
// #3
id: -1,
name: name,
label: label,
items: items
}
}
})
// #4
.catch(err => {
// console.log(err)
})
siteListsPromises.push(promise)
}
b++
}
}
// #5
const siteLists = (await Promise.all(siteListsPromises))
// #6
.filter(el => el !== undefined)
// #7
.map((el, i) => ({ id: i, ...el }))
res.send({ sites: siteLists })
}
res.end()
}
Look for // #N comments in the snippet.
Don't await for requests to complete. Instead iterate over sites and send all requests at once
Chain json() and siteData processing after the fetch with then. And should your processing of siteData be more computational heavy it'd have even more sense to do so, instead of performing all of it only after all promises resolve.
If you (or someone on your team) have some troubles with understanding closures, don't bother setting the id of siteData elements in the cycle. I won't dive in this, but will address it further.
use .catch() instead of try{}catch(){}. Because without await it won't work.
await results of all requests with the Promise.all()
filter out those where siteData was falsy
finally set the id field.

Too many simultaneous requests with NodeJS+request-promise

I have NodeJS project with a BIG array (about 9000 elements) containing URLs. Those URLs are going to be requested using the request-promise package. However, 9000 concurrent GET requests to the same website from the same client is neither liked by the server or the client, so I want to spread them out over time. I have looked around a bit and found Promise.map together with the {concurrency: int} option here, which sounded like it would do what I want. But I cannot get it to work. My code looks like this:
const rp = require('request-promise');
var MongoClient = require('mongodb').MongoClient;
var URLarray = []; //This contains 9000 URLs
function getWebsite(url) {
rp(url)
.then(html => { /* Do some stuff */ })
.catch(err => { console.log(err) });
}
MongoClient.connect('mongodb://localhost:27017/some-database', function (err, client) {
Promise.map(URLArray, (url) => {
db.collection("some-collection").findOne({URL: url}, (err, data) => {
if (err) throw err;
getWebsite(url, (result) => {
if(result != null) {
console.log(result);
}
});
}, {concurrency: 1});
});
I think I probably misunderstand how to deal with promises. In this scenario I would have thought that, with the concurrency option set to 1, each URL in the array would in turn be used in the database search and then passed as a parameter to getWebsite, whose result would be displayed in its callback function. THEN the next element in the array would be processed.
What actually happens is that a few (maybe 10) of the URLs are fetch correctly, then the server starts to respond sporadically with 500 internal server error. After a few seconds, my computer freezes and then restarts (which I guess is due to some kind of panic?).
How can I attack this problem?

If the problem is really about concurrency, you can divide the work into chunks and chain the chunks.
Let's start with a function that does a mongo lookup and a get....
// answer a promise that resolves to data from mongo and a get from the web
// for a given a url, return { mongoResult, webResult }
// (assuming this is what OP wants. the OP appears to discard the mongo result)
//
function lookupAndGet(url) {
// use the promise-returning variant of findOne
let result = {}
return db.collection("some-collection").findOne({URL: url}).then(mongoData => {
result.mongoData = mongoData
return rp(url)
}).then(webData => {
result.webData = webData
return result
})
}
lodash and underscore both offer a chunk method that breaks an array into an array of smaller. Write your own or use theirs.
const _ = require('lodash')
let chunks = _.chunk(URLArray, 5) // say 5 is a reasonable concurrency
Here's the point of the answer, make a chain of chunks so you only perform the smaller size concurrently...
let chain = chunks.reduce((acc, chunk) => {
const chunkPromise = Promise.all(chunk.map(url => lookupAndGet(url)))
return acc.then(chunkPromise)
}, Promise.resolve())
Now execute the chain. The chunk promises will return chunk-sized arrays of results, so your reduced result will be an array of arrays. Fortunately, lodash and underscore both have a method to "flatten" the nested array.
// turn [ url, url, ...] into [ { mongoResult, webResult }, { mongoResult, webResult }, ...]
// running only 5 requests at a time
chain.then(result => {
console.log(_.flatten(result))
})

Pushing elements into the array works only inside the loop

I got some data which I'm calling from API and I am using axios for that. When data is retrieved, I dump it inside of a function called "RefractorData()" just to organize it a bit, then I push it onto existing array. The problems is, my array gets populated inside forEach and I can console.log my data there, but once I exit the loop, my array is empty.
let matches: any = new Array();
const player = new Player();
data.forEach(
async (match: any) => {
try {
const result = await API.httpRequest(
`https://APILink.com/matches/${match.id}`,
false
);
if (!result) console.log("No match info");
const refractored = player.RefractorMatch(result.data);
matches.push({ match: refractored });
console.log(matches);
} catch (err) {
throw err;
}
}
);
console.log(matches);
Now the first console.log inside forEach is displaying data properly, second one after forEach shows empty array.

Managed to do it with Promise.all() and Array.prototype.map()
.
const player = new Player();
const matches = result.data;
const promises = matches.map(async (match: any) => {
const response: any = await API.httpRequest(
`https://API/matches/${match.id}`,
false
);
let data = response.data;
return {
data: player.RefractorMatch(data)
};
});
const response: any = await Promise.all(promises);

You must understand that async functions almost always run later, because they deppend on some external input like a http response, so, the second console.log is running before the first.
There a few ways to solve this. The ugliest but easiest to figure out is to create a external promise that you will resolve once all http requests are done.
let matches = [];
let promise = new Promise((resolve) => {
let complete = 0;
data.forEach((match: any) => {
API.httpRequest(...).then((result) => {
// Your logic here
matches.push(yourLogicResult);
complete++;
if (complete === data.length) {
resolve();
}
}
}
};
console.log(matches); // still logs empty array
promise.then(() => console.log(matches)); // now logs the right array
You can solve this using other methods, for example Promise.all().
One very helpful way to solve it is using RxJs Observables. See https://www.learnrxjs.io/
Hope I helped you!

How to write an arbitrarily long Promise chain

I receive an object bigListFromClient that includes an arbitrary number of objects each of which may have an arbitrary number of children. Every object needs to be entered into my database, but the DB needs to assign each of them a unique ID and child objects need to have the unique ID of their parents attached to them before they are sent off to the DB.
I want to create some sort of Promise or other calling structure that would call itself asynchronously until it reached the last object in bigListFromClient but I'm having trouble figuring out how to write it.
for(let i = 0; i < bigListFromClient.length; i++){
makeDbCallAsPromise(bigListFromClient[i].queryString, console.log); //I'm not just accepting anything from a user here, but how I get my queryString is kind of out of scope for this question
for(let j = 0; j < bigListFromClient[i].children.length; j++){
//the line below obviously doesn't work, I'm trying to figure out how to do this with something other than a for loop
makeDbCallAsPromise(bigListFromClient[i].children[j].queryString + [the uniqueID from the DB to insert this correctly as a child], console.log);
}
}
//this promise works great
makeDbCallAsPromise = function(queryString){
return new Promise((resolve, reject) => {
connection = mysql.createConnection(connectionCredentials);
connection.connect();
query = queryString;
connection.query(query, function (err, rows, fields) {
if (!err) {
resolve(rows);
} else {
console.log('Error while performing Query.');
console.log(err.code);
console.log(err.message);
reject(err);
}
});
connection.end();
})
};
My attempts at solving this on my own are so embarrassingly bad that even describing them to you would be awful.
While I could defer all the calls to creating children until the parents have been created in the DB, I wonder if the approach I've described is possible.

There are essentially two ways to do this. One is making the database calls sequential and the other one is making the calls parallel.
Javascript has a built-in function for parallel called Promise.all, you pass it an array of Promise instances and it returns a Promise instance containing the array.
In your case your code would look like this:
const result = Promise.all(
bigListFromClient.map(item =>
makeDbCallAsPromise(item.queryString).then(result =>
Promise.all(
item.children.map(item =>
makeDbCallAsPromise(item.queryString + [result.someId])
)
)
])
})
result will now contain a Promise that resolves to an array of arrays. These arrays contain the result of intserting children.
Using a more modern approach (with async await), sequential and with all results in a flat array:
const result = await bigListFromClient.reduce(
async (previous, item) => {
const previousResults = await previous
const result = await makeDbCallAsPromise(item.queryString)
const childResults = await item.children.reduce(
async (result, item) =>
[...(await result), await makeDbCallAsPromise(item.queryString + [result.someId])],
[]
)
return [...previousResults, result, ...childResults)
]),
[]
})
Depending on what you want to achieve and how you want to structure your code you can pick and choose from the different approaches.

For this sort of operation, try looking into bulk inserting. If you are intent on performing a single DB query/transaction per iteration, loop recursively over each parent and/or execute the same procedure for each child.
const dbCall = async (elm) => {
elm.id = Math.random().toString(36).substring(7)
if (elm.children) {
await Promise.all(elm.children.map(child => {
child.parentId = elm.id
return dbCall(child)
}))
}
return elm
}
const elms = [
{
queryString: '',
children: [
{
queryString: ''
}
]
}
]
Promise.all(elms.map(dbCall)).then(elm => /* ... */)

node.js: structure multiple API requests, work on them and combine them

currently I am struggeling a little bit with node.js (I am new to it) doing different API requests (Usabilla API), work on the results and then combine them in order to work on the whole set (e.g. export).
Requesting the API is not the problem but I can't get the results out to do some other stuff on it (asynchronous code drives me crazy).
Attached please find a overview how I thought to do this. Maybe I am totally wrong about this or maybe you have other more elegant suggestions.
My code works until I have to request the two different API "adresses" (they are provided) and then extract the results to do some other stuff.
My problem here is that there are nested functions with a promise and I cant figure out how to pass this through the parent function inside waterfall to get handled by the next function.
In the code, of course there is nothing parallel as shown in the diagram.
Thats another point, how to do that ? Simply nest parallel and series/ another waterfall inside waterfall ?
I am a little bit confused because that gets more and more complex for a simple problem when this would be done with synchronous code.
Here I build up all my request querys (at the moment 4):
function buildQuery(IDs,callback){
var i = 0;
var max = Object.keys(IDs).length;
async.whilst(
function(){return i < max},
function(callback){
FeedbackQuery[i] =
{
identifier: IDs[i].identifier,
query:
{id: IDs[i].id,
params: {since:sinceDate,}
}
};
i++;
callback(null,i);
})
console.log(FeedbackQuery);
callback (null,FeedbackQuery);
};
I then have to decide which type of query it is and add it to an object which should contain all the items of this identifier type:
function FeedbackRequest(FeedbackQuery,callback)
{
var i = 0;
var max = Object.keys(FeedbackQuery).length;
async.whilst(
function(){return i < max},
function (callback){
identifier = FeedbackQuery[i].identifier;
APIquery = FeedbackQuery[i].query;
switch(identifier)
{
case 'mobilePortal':
console.log(FeedbackQuery[i].identifier, 'aktiviert!');
var result = api.websites.buttons.feedback.get(APIquery);
result.then(function(feedback)
{
var item = Object.keys(feedbackResults).length;
feedbackResultsA[item] = feedback;
callback(null, feedbackResultsA);
})
break;
case 'apps':
console.log(FeedbackQuery[i].identifier, 'aktiviert!');
var result = api.apps.forms.feedback.get(APIquery);
result.then(function(feedback)
{
var item = Object.keys(feedbackResults).length;
feedbackResultsB[item] = feedback;
callback(null, feedbackResultsB);
})
break;
}
i++;
callback(null,i);
})
};
Currently the functions are bundled in an async waterfall:
async.waterfall([
async.apply(buildQuery,IDs2request),
FeedbackRequest,
// a function to do something on the whole feedbackResults array
],function (err, result) {
// result now equals 'done'
if (err) { console.log('Something is wrong!'); }
return console.log('Done!');
})
How it actually should be:
Structure
Thank you very much for any tips or hints!

I'm not proficient with async, and I believe if you'r new to this, it's harder than a simple Promise library like bluebird combine with lodash for helpers.
What I would do based on your schemas :
var firstStepRequests = [];
firstStepRequests.push(buildQuery());// construct your first steps queries, can be a loop, goal is to have firstStepRequests to be an array of promise.
Promise.all(firstStepRequests)
.then((allResults) => {
var type1 = _.filter(allResults, 'request_type_1');
var type2 = _.filter(allResults, 'request_type_2');
return {
type1: type1,
type2: type2
};
})
.then((result) => {
result.type1 = //do some work
result.type2 = //do some work
return result;
})
.then((result) => {
//export or merge or whatever.
});
Goal is to have a simple state machine.
UPDATE
If you want to keep identifier for a request, you can use props to have :
var props = {
id_1:Promise,
id_2:Promise,
id_3:Promise
};
Promise.props(props).then((results) => {
// results is {
id_1:result of promise,
id_2:result of promise,
etc...
}
})
You could do something like :
var type1Promises = getType1Requests(); //array of type 1
var type2Promises = getType2Requests(); // array of type 2
var props = {
type_1: Promise.all(type1Promises),
type_2: Promise.all(type2Promises)
}
Promise.props(props).then((result) => {
//result is : {
type_1: array of result of promises of type 1
type_2: array of result of promises of type 2
}
})