Develop Reference

How to achieve recursive Promise calls in Node.js

I am calling an API where I can only fetch 1000 records per request,
I was able to achieve this using recursion.
I am now trying to achieve the same using promises, I am fairly new to Node.js and JavaScript too.
I tried adding the recursion code in an if else block but failed
var requestP = require('request-promise');
const option = {
url: 'rest/api/2/search',
json: true,
qs: {
//jql: "project in (FLAGPS)",
}
}
const callback = (body) => {
// some code
.
.
.//saving records to file
.
//some code
if (totlExtractedRecords < total) {
requestP(option, callback).auth('api-reader', token, true)
.then(callback)
.catch((err) => {
console.log('Error Observed ' + err)
})
}
}
requestP(option).auth('api-reader', token, true)
.then(callback)
.catch((err) => {
console.log('Error Observed ' + err)
})
I want to execute the method using promise and in a synchronous way,
i.e. I want to wait until the records are all exported to a file and continue with my code

I think its better to create your own promise and simply resolve it when your done with your recursion. Here's a simply example just for you to understand the approach
async function myRecursiveLogic(resolveMethod, ctr = 0) {
// This is where you do the logic
await new Promise((res) => setTimeout(res, 1000)); // wait - just for example
ctr++;
console.log('counter:', ctr);
if (ctr === 5) {
resolveMethod(); // Work done, resolve the promise
} else {
await myRecursiveLogic(resolveMethod, ctr); // recursion - continue work
}
}
// Run the method with a single promise
new Promise((res) => myRecursiveLogic(res)).then(r => console.log('done'));

Here's a clean and nice solution using the latest NodeJS features.
The recursive function will continue executing until a specific condition is met (in this example asynchronously getting some data).
const sleep = require('util').promisify(setTimeout)
const recursive = async () => {
await sleep(1000)
const data = await getDataViaPromise() // you can replace this with request-promise
if (!data) {
return recursive() // call the function again
}
return data // job done, return the data
}
The recursive function can be used as follows:
const main = async () => {
const data = await recursive()
// do something here with the data
}

Using your code, I'd refactored it as shown below. I hope it helps.
const requestP = require('request-promise');
const option = {
url: 'rest/api/2/search',
json: true,
qs: {
//jql: "project in (FLAGPS)",
}
};
/*
NOTE: Add async to the function so you can udse await inside the function
*/
const callback = async (body) => {
// some code
//saving records to file
//some code
try {
const result = await requestP(option, callback).auth('api-reader', token, true);
if (totlExtractedRecords < total) {
return callback(result);
}
return result;
} catch (error) {
console.log('Error Observed ' + err);
return error;
}
}

Created this code using feed back from Amir Popovich
const rp = require('Request-Promise')
const fs = require('fs')
const pageSize = 200
const options = {
url: 'https://jira.xyz.com/rest/api/2/search',
json: true,
qs: {
jql: "project in (PROJECT_XYZ)",
maxResults: pageSize,
startAt: 0,
fields: '*all'
},
auth: {
user: 'api-reader',
pass: '<token>',
sendImmediately: true
}
}
const updateCSV = (elment) => {
//fs.writeFileSync('issuedata.json', JSON.stringify(elment.body, undefined, 4))
}
async function getPageinatedData(resolve, reject, ctr = 0) {
var total = 0
await rp(options).then((body) => {
let a = body.issues
console.log(a)
a.forEach(element => {
console.log(element)
//updateCSV(element)
});
total = body.total
}).catch((error) => {
reject(error)
return
})
ctr = ctr + pageSize
options.qs.startAt = ctr
if (ctr >= total) {
resolve();
} else {
await getPageinatedData(resolve, reject, ctr);
}
}
new Promise((resolve, reject) => getPageinatedData(resolve, reject))
.then(() => console.log('DONE'))
.catch((error) => console.log('Error observed - ' + error.name + '\n' + 'Error Code - ' + error.statusCode));

How do I call 3 requests async?

I have to do a functionality to test if 3 APIs are running.
Thus, the user will click on the Test APIs button and it will return the status of each API (status: 200, 500, 404 etc). If an API return an error, I should show the error stack.
Screen example:
API Status Detail
url1.com 200 -
url2.com 200 -
url3.com 500 internal server error
My question is, how can I call the 3 requests in parallel and return the async result, I mean how can I update the screen of API request status without having to wait for the result of all requests
I was basing on that How do I call three requests in order?, but it returns the result synchronously.
*******EDIT*****
Thats my current code
app.get('/testDependencies', function (req, res, next) {
let objTestsResul = {}
var urls = ['url1', 'url2', 'url3'];
let index = 0
while(urls.length > 0) {
let url = urls.shift();
objTestsResult[index++] = testURL(url)
}
res.send(objTestsResult)
});
This function is the same for each URL:
function testURL(URL){
fetch(URL, {
method: 'GET'
})
.then(res => {
res.json()
})
.then(json => {
console.log(json)
return json
})
.catch(error => {
return error
})
}

Promises (mdn) seem to be what you're looking for. They're essentially a more readable version of callbacks, which allow you to execute code when something else occurs rather than having to wait for that trigger to occur before resuming execution.
let endpoint1 = () => new Promise(resolve => setTimeout(() => resolve('200'), 1000));
let endpoint2 = () => new Promise(resolve => setTimeout(() => resolve('201'), 2000));
let endpoint3 = () => new Promise(resolve => setTimeout(() => resolve('500'), 1500));
document.getElementById('test').addEventListener('click', () => {
document.getElementById('status').textContent = 'test running...';
Promise.all([
endpoint1().then(a => document.getElementById('result1').textContent = a),
endpoint2().then(a => document.getElementById('result2').textContent = a),
endpoint3().then(a => document.getElementById('result3').textContent = a),
]).then(() => document.getElementById('status').textContent = 'test complete');
});
<button id="test">test</button>
<div>status: <span id="status">not running</span></div>
<div>endpoint 1: <span id="result1"></span></div>
<div>endpoint 2: <span id="result2"></span></div>
<div>endpoint 3: <span id="result3"></span></div>

This is actually pretty straightforward if you can use Bluebird:
const { Promise } = require('bluebird');
app.get('/testDependencies', function (req, res, next) {
Promise.map(['url1', 'url2', 'url3'], url => testURL(url)).then(results => {
res.send(results);
});
});
You'll just need to ensure your promise function actually returns a promise:
function testURL(URL) {
let start_time = new Date().getTime();
return fetch(URL, {
method: 'GET'
}).then(res => {
res.json()
}).then(json => {
console.log(json)
return json
}).catch(error => {
return error
})
}
Promises can't be dependency chained unless you explicitly return them from the function that's involved in chaining.
If you're able to use async and await, I'd also recommend doing that as well as that can vastly simplify otherwise complex code.

Express can't send multiple responses. You will have to finish all calls or use WebSockets to stream data.
function testURL(URL) {
return new Promise((resolve, reject) => {
if (URL === 'url2') {
reject(new Error('Internal Server Error'));
return;
}
resolve({ status: 200 });
});
}
const main = async () => {
const urls = ['url1', 'url2', 'url3'];
// return resolved and rejected Promises because if one fails in Promise.all
// the function will throw and we won't have any access to any resolved Promises.
const results = await Promise.all(urls
.map(url => testURL(url).then(response => response).catch(error => error)));
// every error have a stack property, Set the status to whatever you want
// based on the error and store the stack and the message
const objTestsResul = results.reduce((result, cur, i) => {
result[urls[i]] = cur.stack
? { status: 500, message: cur.message, stack: cur.stack }
: cur;
return result;
}, {});
console.log(objTestsResul);
};
main();

Calling async function multiple times

So I have a method, which I want to call multiple times in a loop. This is the function:
function PageSpeedCall(callback) {
var pagespeedCall = `https://www.googleapis.com/pagespeedonline/v4/runPagespeed?url=https://${websites[0]}&strategy=mobile&key=${keys.pageSpeed}`;
// second call
var results = '';
https.get(pagespeedCall, resource => {
resource.setEncoding('utf8');
resource.on('data', data => {
results += data;
});
resource.on('end', () => {
callback(null, results);
});
resource.on('error', err => {
callback(err);
});
});
// callback(null, );
}
As you see this is an async function that calls the PageSpeed API. It then gets the response thanks to the callback and renders it in the view. Now how do I get this to be work in a for/while loop? For example
function PageSpeedCall(websites, i, callback) {
var pagespeedCall = `https://www.googleapis.com/pagespeedonline/v4/runPagespeed?url=https://${websites[i]}&strategy=mobile&key=${keys.pageSpeed}`;
// second call
var results = '';
https.get(pagespeedCall, resource => {
resource.setEncoding('utf8');
resource.on('data', data => {
results += data;
});
resource.on('end', () => {
callback(null, results);
});
resource.on('error', err => {
callback(err);
});
});
// callback(null, );
}
var websites = ['google.com','facebook.com','stackoverflow.com'];
for (let i = 0; i < websites.length; i++) {
PageSpeedCall(websites, i);
}
I want to get a raport for each of these sites. The length of the array will change depending on what the user does.
I am using async.parallel to call the functions like this:
let freeReportCalls = [PageSpeedCall, MozCall, AlexaCall];
async.parallel(freeReportCalls, (err, results) => {
if (err) {
console.log(err);
} else {
res.render('reports/report', {
title: 'Report',
// bw: JSON.parse(results[0]),
ps: JSON.parse(results[0]),
moz: JSON.parse(results[1]),
// pst: results[0],
// mozt: results[1],
// bw: results[1],
al: JSON.parse(results[2]),
user: req.user,
});
}
});
I tried to use promise chaining, but for some reason I cannot put it together in my head. This is my attempt.
return Promise.all([PageSpeedCall,MozCall,AlexaCall]).then(([ps,mz,al]) => {
if (awaiting != null)
var areAwaiting = true;
res.render('admin/', {
title: 'Report',
// bw: JSON.parse(results[0]),
ps: JSON.parse(results[0]),
moz: JSON.parse(results[1]),
// pst: results[0],
// mozt: results[1],
// bw: results[1],
al: JSON.parse(results[2]),
user: req.user,
});
}).catch(e => {
console.error(e)
});
I tried doing this:
return Promise.all([for(let i = 0;i < websites.length;i++){PageSpeedCall(websites, i)}, MozCall, AlexaCall]).
then(([ps, mz, al]) => {
if (awaiting != null)
var areAwaiting = true;
res.render('admin/', {
title: 'Report',
// bw: JSON.parse(results[0]),
ps: JSON.parse(results[0]),
moz: JSON.parse(results[1]),
// pst: results[0],
// mozt: results[1],
// bw: results[1],
al: JSON.parse(results[2]),
user: req.user,
});
}).catch(e => {
console.error(e)
});
But node just said it's stupid.
And this would work if I didn't want to pass the websites and the iterator into the functions. Any idea how to solve this?
To recap. So far the functions work for single websites. I'd like them to work for an array of websites.
I'm basically not sure how to call them, and how to return the responses.

It's much easier if you use fetch and async/await
const fetch = require('node-fetch');
async function PageSpeedCall(website) {
const pagespeedCall = `https://www.googleapis.com/pagespeedonline/v4/runPagespeed?url=https://${website}&strategy=mobile&key=${keys.pageSpeed}`;
const result = await fetch(pagespeeddCall);
return await result.json();
}
async function callAllSites (websites) {
const results = [];
for (const website of websites) {
results.push(await PageSpeedCall(website));
}
return results;
}
callAllSites(['google.com','facebook.com','stackoverflow.com'])
.then(results => console.log(results))
.error(error => console.error(error));
Which is better with a Promise.all
async function callAllSites (websites) {
return await Promise.all(websites.map(website => PageSpeedCall(website));
}

Starting on Node 7.5.0 you can use native async/await:
async function PageSpeedCall(website) {
var pagespeedCall = `https://www.googleapis.com/pagespeedonline/v4/runPagespeed?url=https://${website}&strategy=mobile&key=${keys.pageSpeed}`;
return await promisify(pagespeedCall);
}
async function getResults(){
const websites = ['google.com','facebook.com','stackoverflow.com'];
return websites.map(website => {
try {
return await PageSpeedCall(website);
}
catch (ex) {
// handle exception
}
})
}
Node http "callback" to promise function:
function promisify(url) {
// return new pending promise
return new Promise((resolve, reject) => {
// select http or https module, depending on reqested url
const lib = url.startsWith('https') ? require('https') : require('http');
const request = lib.get(url, (response) => {
// handle http errors
if (response.statusCode < 200 || response.statusCode > 299) {
reject(new Error('Failed to load page, status code: ' + response.statusCode));
}
// temporary data holder
const body = [];
// on every content chunk, push it to the data array
response.on('data', (chunk) => body.push(chunk));
// we are done, resolve promise with those joined chunks
response.on('end', () => resolve(body.join('')));
});
// handle connection errors of the request
request.on('error', (err) => reject(err))
})
}

Make PageSpeedCall a promise and push that promise to an array as many times as you need, e.g. myArray.push(PageSpeedCall(foo)) then myArray.push(PageSpeedCall(foo2)) and so on. Then you Promise.all the array.
If subsequent asynch calls require the result of a prior asynch call, that is what .then is for.
Promise.all()
Promise.all([promise1, promise2, promise3]).then(function(values) {
console.log(values);
});

how can I return the array from this promise?

I have tried a few methods and been reading round but I cannot seem to figure out how to return the names array from this function.
function getNames(oauth2Client, docs) {
const api = x('v1');
let names = [];
return Promise.each(docs, function(doc) {
let req = api.users.messages.get;
let options = ({
auth: oauth2Client,
'userId': 'me',
'id': doc.id
});
return Promise.promisify(req)(options).then(function(response) {
for (y = 0; y < response.names.length; y++) {
names.push(response.names[y].toLowerCase());
}
})
.catch(function (err) {
console.log('An error occured: ' + err.message);
throw err;
});
});
}

I'm not sure what Promise library you are using as it appears non-standard, but something like this I think is what you want. I added comments for what's going on - you might have to change those lines of code to suit your promise library.
function getNames(oauth2Client, docs) {
const api = x('v1');
const names = [];
// create a stack of promises
const stack = [];
docs.forEach(doc => {
let req = api.users.messages.get;
let options = ({
auth: oauth2Client,
'userId': 'me',
'id': doc.id
});
// push each promise onto the stack
stack.push(
Promise.promisify(req)(options).then(function(response) {
for (y = 0; y < response.names.length; y++) {
names.push(response.names[y].toLowerCase());
}
})
.catch(function (err) {
console.log('An error occured: ' + err.message);
throw err;
})
);
});
// Wait for all promises in the stack to finish, and then
// return the names array as the final value.
return Promise.all(stack).then(() => names);
}

Simply add
return Promise.each(…)
.then(function() {
return names;
});
that causes the returned promise to fulfill with the names array.
However I would recommend that you don't use a global array across the each loop, especially if you care about the order of results. Instead, resolve every promise with a value, use map instead of each, and combine the results in the end:
const api = x('v1');
const getUserMessages = Promise.promisify(api.users.messages.get);
function getNames(oauth2Client, docs) {
return Promise.map(docs, doc =>
getUserMessages({
auth: oauth2Client,
'userId': 'me',
'id': doc.id
})
.then(response =>
response.names.map(name => name.toLowerCase());
)
)
.then(nameArrays =>
[].concat(...nameArrays)
);
}

Promises are not behaving as I expect them to

I'm using Express for routing and Sequelize for DB management.
app.get('/api/users/:username', (req, res) => {
let username = req.params.username;
findChattersPerRole()
.then(chattersPerRole => {
console.log('instakbot should\'ve been added by now...');
});
});
The function findChattersPerRole returns an object with each user's username and role as another object.
const findChattersPerRole = () => {
return fetch('https://tmi.twitch.tv/group/user/instak/chatters')
.then(parseJSON)
.then(r => {
let chatters = r.chatters;
let chattersPerRole = Object.keys(chatters).map(role => {
return chatters[role].map(username => {
console.log('findOrCreateViewer will be executed after this');
findOrCreateViewer(username, role);
return {
username: username,
role: role
};
});
});
return Promise.resolve(flattenDeep(chattersPerRole));
}).catch(err => {
console.log(`Error in fetch: ${err}`);
});
};
The problem is, in my route, I expect the console.log('instakbot should\'ve been added by now...'); to be executed AFTER my viewers got inserted into the database because in my function findChattersPerRole I already insert them with the function findOrCreateViewer. I expect this to happen because in my route I write the console.log when findChattersPerRole() is resolved...
const findOrCreateViewer = (username, role) => {
return Viewer.findOrCreate({
where: {
username
},
defaults: {
instakluiten: 5,
role
}
}).spread((unit, created) => {
console.log('unit is: ', unit.dataValues.username);
if(created){
return `created is ${created}`;
}else{
return unit;
}
});
};
However, in my terminal you can see that this is not the way it's happening... Why aren't my promises being executed at the expected time?
Screenshot of my terminal

The return {username: ...} after findOrCreateViewer(username, role); happens immediately after the function is called and before any data has been inserted. That also means that return Promise.resolve(flattenDeep(chattersPerRole)); happens before any data has been inserted, etc.
You said findOrCreateViewer returns a promise, so you need to wait until that promise is resolved (i.e. wait until after the data was inserted) before continuing with something else.
You want chattersPerRole to be an array of (arrays of) promises and only proceed after all the promises are resolved.
This is easy to do with Promise.all:
const findChattersPerRole = () => {
return fetch('https://tmi.twitch.tv/group/user/instak/chatters')
.then(parseJSON)
.then(r => {
let chatters = r.chatters;
let chattersPerRole = Object.keys(chatters).map(
role => chatters[role].map(username => {
console.log('findOrCreateViewer will be executed after this');
return findOrCreateViewer(username, role).then(
() => ({username, role})
);
});
);
return Promise.all(flattenDeep(chattersPerRole));
}).catch(err => {
console.log(`Error in fetch: ${err}`);
});
};
Now the promise returned by findChattersPerRole will be resolved after all the promises returned by findOrCreateViewer are resolved.

Promises are doing no magic. Returning a promise doesn't mean that calling the function will block, but rather that you can easily chain callbacks to do something with the result. You'll need to use
function findChattersPerRole() {
return fetch('https://tmi.twitch.tv/group/user/instak/chatters')
.then(parseJSON)
.then(r => {
let chatters = r.chatters;
let chattersPerRole = Object.keys(chatters).map(role => {
return chatters[role].map(username => {
console.log('findOrCreateViewer will be executed after this');
return findOrCreateViewer(username, role).then(() => {
// ^^^^^^ ^^^^^
return {
username: username,
role: role
};
});
});
});
return Promise.all(flattenDeep(chattersPerRole));
// ^^^ get a promise for an array of results from an array of promises
}).catch(err => {
console.log(`Error in fetch: ${err}`);
});
}