Develop Reference

How to return a Promise with consequential axios calls?

I need to create a function, that will return a Promise and will call another function that will have an axios.get() call. Axios.get() calls an API that returns a data with the following structure:
{
count: 87, //total number of records
next: '[api_url]/&page=2'//indication if the API has a next page
previous: null, ////indication if the API has a previous page, if call URL above - it will return [api_url]/&page=1
results: [...] //10 objects for each call, or remaining for the last page
}
Since I know that only 10 results are being returned on every call, I need to check if the returned object has the next key, and if it does - make another call and so on, until no more next. I need to concatenate all the results and eventually resolve the Promise returned from the main function with all the data.
So I tried something like that:
const fetchResource = async({type, search, page}) {
const data = {count: 0, results: []}
const request = await performFetch({type, search, page}, data).then((data) => {
console.log('data?', data)
})
console.log('req', request)
}
const performFetch = async({type, search, page}, result) => {
const params = {
page
}
if (search) {
params.search = search
}
await axios.get(`${type}/`, {
params
}).then(async({data}) => {
result.results = [...result.results, ...data.results]
result.count = data.count
if (data.next) {
page += 1
await performFetch({type, search, page}, result)
} else {
console.log('result', result)
return result
}
})
.catch((err) => {
console.error(err)
})
}
Now I see that once I call fetchResourche all the requests are going out, and in console.log('result', result) I do see the concatenated data:
{
count: 87,
results: [/*all 87 objects*/]
}
But console.log('data?', data) and console.log('req', request) both print out undefined.
Where I return result, I tried to return Promise.resolve(result) - same result.
And I'm not sure how to return a Promise here, that will resolve once all the API calls are concluded and all the data is received. What am I missing? How do I make it work?

Couple of observations regarding your code:
There's no need to mix async-await syntax with promise chaining, i.e. then() and catch() method calls
Inside performFetch function, you need an explicit return statement. Currently, the function is implicitly returning a promise that fulfils with the value of undefined.
Key point here is that the performFetch function is returning before you get the result of http requests to the API.
It isn't waiting for the result of HTTP requests to be returned before returning.
Following is a simple demo that illustrates how you can make multiple requests and aggregate the data until API has returned all the data.
let counter = 0;
function fakeAPI() {
return new Promise(resolve => {
setTimeout(() => {
if (counter < 5) resolve({ counter: counter++ });
else resolve({ done: true });
}, 1000);
});
}
async function performFetch() {
const results = [];
// keep calling the `fakeAPI` function
// until it returns "{ done = true }"
while (true) {
const result = await fakeAPI();
if (result.done) break;
else results.push(result);
}
return results;
}
performFetch().then(console.log).catch(console.log);
<small>Wait for 5 seconds</small>
Your code can be rewritten as shown below:
const fetchResource = async ({ type, search, page }) => {
const data = { count: 0, results: [] };
const result = await performFetch({ type, search, page }, data);
console.log(result);
};
const performFetch = async ({ type, search, page }, result) => {
const params = { page };
if (search) params.search = search;
while (true) {
const { data } = await axios.get(`${type}/`, { params });
result.results = [...result.results, ...data.results];
result.count = data.count;
if (data.next) page += 1;
else return result;
}
};
Ideally, the code that calls the fetchResource function should do the error handling in case any of the HTTP request fails.

How would initiate requests to an array of links, and then after they have all resolved, render results using Express?

I am trying to use axios and express to get an array of links from a page, harvest data at each link, and display results to the user. The process I'd like to implement is:
Run axios.get(targetPage), harvest the links on the target page, save to an array
Run run axios.all to get a response from link
Harvest data from each response
Display to user
Below is my app.get function:
app.get('/search', function (req, res) {
var context = {};
context.resources = [];
var promises = [];
var year = req.query.year;
var targetPage = endpoint + year;
axios.get(targetPage).then(resp => {
var $ = cheerio.load(resp.data);
var pages = []
$('queryStr').each(function (i, ele) { pages.push(endpoint + $(ele).attr("href")) });
context.links = pages;
pages.forEach( link => promises.push( axios.get(link) ));
}).then(axios.all(promises)).then( responses => {
responses.forEach( resp => {
var resource = {};
resource.link = resp.url;
var $ = cheerio.load(resp.data)
resource.title = $('query').text()
context.resources.push(resource);
})
}).then( () => {
res.render('search', context);
})
})
I have verified that the urls in the pages[] array are valid. I have tried calling res.render after the first axios.get call, and it successfully rendered my response. I have separately tested the logic in the forEach and verified that it works for each url. I'm getting an error message at the then block immediately following axios.all, which states that responses returned by axios.all is undefined. the Here is the error message:
UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'forEach' of undefined
Thanks for reading.

[TLDR]
This is because the function of your first then clause doesn't have a return statement.
This means there is no successful data passed to the next then clause and so on.
then chaining requires return for each then clause.
Example

There's many problems in my original post, but the main issue was failure to return a Promise in each step of my .then chain. Below is working code, significantly refactored. I'm sure there's better ways of passing along errors and loading in parallel, so please leave answers and comments with any improvements. In addition to the resource linked to by Nazim Kerimbekov and Khoa, I recommend this post for learning how to chain promises.
app.get('/pageToRender', function (req, res) {
var context = {};
var page1 = req.query.year
getInitial(page1)
.then( pages => Promise.all( pages.map( page => { return getResource(page) })))
.then(resources => {
context.resources = resources;
res.render('pageToRender', context);
})
.catch( errors => { console.log(errors) })
})
function getInitial (page) {
return new Promise( (resolve,reject) => {
axios.get(page).then( resp => {
var pages = []
var $ = cheerio.load(resp.data)
$('.mw-content-ltr li a').each(function (i, ele) { pages.push(endpoint + $(ele).attr("href")) });
console.log(pages);
resolve(pages);
}).catch( error => reject(error))
})
}
function getResource(page) {
return new Promise ( (resolve, reject) => {
axios.get(page)
.then( response => {
var resource = {};
resource.link = page;
var $ = cheerio.load(response.data)
resource.title = $('.Title').text()
console.log("resolving resource", resource);
resolve(resource);
}).catch (error => { error })
})
}

Async functions using value from a promise

So I know this question is asked a lot, but I'm trying to retrieve a variable that is created within a promise. The examples I've seen on here involve calling .then and using the data there, however what I'm trying to do involves an async function--which i cant use within the .then block.
Here's my code. I'm using the Asana API To call out a lists of tasks that are due. It successfuly logs it. But I want to save the list value from the last block as a variable that I can use elsewhere.
const asana = require('asana');
const client = asana.Client.create().useAccessToken("xxx");
client.users.me()
.then(user => {
const userId = user.id;
// The user's "default" workspace is the first one in the list, though
// any user can have multiple workspaces so you can't always assume this
// is the one you want to work with.
const workspaceId = user.workspaces[0].id;
return client.tasks.findAll({
assignee: userId,
workspace: workspaceId,
completed_since: 'now',
opt_fields: 'id,name,assignee_status,completed'
});
})
.then(response => {
// There may be more pages of data, we could stream or return a promise
// to request those here - for now, let's just return the first page
// of items.
return response.data;
})
.filter(task => {
return task.assignee_status === 'today' ||
task.assignee_status === 'new';
})
.then(list => {
console.log (util.inspect(list, {
colors: true,
depth: null
}));
})
.catch(e => {
console.log(e);
});

If you're open to rewriting your .then()'s as async/await something like this could work for you:
const fetch = require('node-fetch');
async function doit() {
const response = await fetch('https://jsonplaceholder.typicode.com/todos/1');
const json = await response.json();
console.log(json);
}
doit();

async.queue within a promise chain?

I am trying to create an async queue for an array of get requests to an api, i am just unsure how to combine and use the responses. Maybe my implementation is wrong since i am using async.queue inside a promise then function ?
Ultimately i would like to get results from first promise ->
use results of that first promise to create an array of get requests for the async.queue ->
then combine the results of all the get responses. I need to throttle the amount of requests that go out at a time due to API rate limit.
const rp = require("request-promise");
app.get("/", (req,res) => {
let arr = []
rp.get(url)
.then((response) => {
let arrayID = response
let q = async.queue((task, callback) => {
request({
method: "GET",
url: url,
qs: {
id: task.id
}
}, (error, response, body) => {
arr.push(body)
console.log(arr.length)
// successfully gives me the response i want. im trying to push into an array with all of my responses,
// but when i go to next then chain it is gone or if i try to return arr i get an empty []
})
callback()
}, 3)
for(var i = 0; i < arrayID.length; i++){
q.push({ id : arrayID[i]} );
}
q.drain = function() {
console.log('all items have been processed');
}
return arr
})
.then((responseArray) => {
//empty array even though the length inside the queue said other wise, i know its a problem with async and sync actions but is there a way to make the promise chain and async queue play nice?
res.json(responseArray)
})
})

Figured it out, ended up having to wrap it in a promise and resolve the final array in q.drain()
const rp = require("request-promise");
app.get("/", (req,res) => {
rp.get(url)
.then((response) => {
let arrayID = response
return new Promise((resolve, reject) => {
var q = async.queue((task, callback) => {
request({
method: "GET",
url: url,
qs: {
id:task.id,
},
}, (error, response, body) => {
arr.push(body)
callback();
})
}, 2);
q.drain = () => resolve(arr);
q.push(arrayID);
})
})
.then((response) => res.json(response))
.catch((error) => res.json(error))
}

To launch multiple async calls in parallel you can use Promise.all()
To launch multiple async calls sequentially (i.e they depend on each other) you can return each promise and use its result inside a then() function
Code below:
app.get("/", (req,res)
.then(function(firstResult)) {
//You can use result of first promise here
return Promise.all([
//Create array of get request here
//To also return firstResult just add it in the Promise.All array
]);
})
.then(function(allResults){
//You can use results of all the get requests created in the previous then()
})
.catch(function(error){
//Deal with any error that happened
});

chaining promises to force async

I'm using promises to fetch large albums of images and when pull random samples from that album. I have managed to request all the albums and then push the links to images to an array of objects.
Now I want to print out that array but only after I've actually filled it. Whenever I add a .then() on the end it prints out only the initialized empty array.
What can I do to force async and only print the array once it's filled. (I'm printing it out at the bottom)
let findImagesCatalyst = new Promise(function(resolve, reject) {
//url options
const options = {
url: 'https://api.imgur.com/3/gallery/hot/time/',
headers: {
"Authorization": "Client-ID xxxx"
}
};
//inital request
request(options, function(err, res, body) {
//parse the response
body = JSON.parse(body)
//access the data in the response
const responseData = body.data;
//filter only those with image counts great than 50
const largeAlbums = responseData.filter(findDumps)
//test to see if a dump is present
if (largeAlbums.length > 0) {
largeAlbums.forEach(function(i) {})
resolve(largeAlbums)
} else {
reject()
}
})
})
//if successful in finding a dump, then go through them and find their albumIds
.then(function(largeAlbums) {
let dumpIds = largeAlbums.map(index => index.id)
return dumpIds;
})
//with the album/dump ids, get each of them with a new request
.then(function(dumpIds) {
//for each of the dumpIds create the needed url using ES6 and then request it.
dumpIds.forEach(function(i) {
const albumUrlOptions = {
url: `https://api.imgur.com/3/album/${i}/images`,
headers: {
"Authorization": "Client-ID xxxx"
}
}
//make a request to each of the albums/dumps
request(albumUrlOptions, function(err, res, body) {
body = JSON.parse(body)
const responseData = body.data
//pick one sample image from the album/dump
let sampleImage = responseData[randomSelector(responseData.length)].link;
dumps.push({
"dump": i,
'sample': sampleImage
})
})
})
return dumps;
})
.then(function(dumps) {
console.log(dumps)
})

You're second .then should return Promise.all of the (promisified) requests
.then(function(dumpIds) {
//for each of the dumpIds create the needed url using ES6 and then request it.
return Promise.all(dumpIds.map(function(i) {
const albumUrlOptions = {
url: `https://api.imgur.com/3/album/${i}/images`,
headers: {
"Authorization": "Client-ID xxxx"
}
};
return new Promise((resolve, reject) => {
//make a request to each of the albums/dumps
request(albumUrlOptions, function(err, res, body) {
body = JSON.parse(body)
const responseData = body.data
//pick one sample image from the album/dump
let sampleImage = responseData[randomSelector(responseData.length)].link;
resolve({
"dump": i,
'sample': sampleImage
});
});
});
}))
})
As you are using node.js, which has very good ES2015+ implementation, you can simplify (in my opinion) your code by, firstly, creating a "promisified version of request
let requestP = (options) => new Promise((resolve, reject) => {
request(options, (err, res, body) => {
if (err) {
return reject(err);
}
resolve({res, body});
});
});
The rest of the code could be then re-written as follows
const options = {
url: 'https://api.imgur.com/3/gallery/hot/time/',
headers: {
"Authorization": "Client-ID xxxx"
}
};
//inital request
let findImagesCatalyst = requestP(options)
.then(({res, body}) => {
//parse the response
body = JSON.parse(body)
//access the data in the response
const responseData = body.data;
//filter only those with image counts great than 50
const largeAlbums = responseData.filter(findDumps)
//test to see if a dump is present
if (largeAlbums.length > 0) {
largeAlbums.forEach(function(i) {})
return(largeAlbums)
} else {
return Promise.reject();
}
})
//if successful in finding a dump, then go through them and find their albumIds
.then((largeAlbums) => largeAlbums.map(index => index.id))
//with the album/dump ids, get each of them with a new request
.then((dumpIds) =>
//for each of the dumpIds create the needed url using ES6 and then request it.
Promise.all(dumpIds.map((i) => {
const albumUrlOptions = {
url: `https://api.imgur.com/3/album/${i}/images`,
headers: {
"Authorization": "Client-ID xxxx"
}
};
return requestP(albumUrlOptions)
.then(({res, body}) => {
body = JSON.parse(body)
const responseData = body.data
//pick one sample image from the album/dump
let sampleImage = responseData[randomSelector(responseData.length)].link;
return({
"dump": i,
'sample': sampleImage
});
});
}))
)
.then(function(dumps) {
console.log(dumps)
});

So, you have a few building blocks here:
Request for imgur albums reflected into options object.
findDumps — a simple function that you filter the list of albums against.
A function that applies the preceding two and returns an array of large albums. It's an asynchronous function, so it likely employs Promise.
A function that takes every item of the array of large albums and receives a single image. It's an asynchronous function, so, again, a Promise.
You want to wait until all the single images have been received.
Finally, you expect an array of objects of two properties: "dump" and "sample".
Let's try to contruct an example.
const findImagesCatalyst = new Promise((resolveImagesCatalyst, rejectImagesCatalyst) => {
const options = {
url: 'https://api.imgur.com/3/gallery/hot/time/',
headers: {
Authorization: 'Client-ID xxxx'
}
};
request(options, (err, res, body) => {
//access the data in the response
const responseData = JSON.parse(body).data;
//filter only those with image counts great than 50
const largeAlbums = responseData.filter(findDumps);
//test to see if a dump is present
if (largeAlbums.length > 0) {
// /!\ The trickiest part here: we won't resolve this promise until an "inner Promise" has been resolved.
// Note that next line declares a new function to resolve inner Promise, resolveLargeAlbum. Now we have two functions:
// - resolveImagesCatalyst - to resolve the main Promise, and
// - resolveLargeAlbum — to resolve every image request, and there can be many of them.
const imagesPromises = largeAlbums.map(largeAlbum => new Promise((resolveLargeAlbum, rejectLargeAlbun) => {
// take id from every large album
const dumpId = largeAlbum.id;
// declare options for inner request
const options = {
url: `https://api.imgur.com/3/album/${i}/images`,
headers: {
"Authorization": "Client-ID xxxx"
}
};
request(albumUrlOptions, (err, res, body) => {
const responseData = JSON.parse(body).data;
//pick one sample image from the album/dump
const sampleImage = responseData[randomSelector(responseData.length)].link;
if (sampleImage) {
// A-HA!
// It's inner Promise's resolve function. For N albums, there will be N resolveLargeAlbum calls. Just a few lines below, we're waiting for all of them to get resolved.
resolveLargeAlbum({
dump: dumpId,
sample: sampleImage
});
} else {
rejectLargeAlbun('Sorry, could not receive sample image:', dumpId, responseData);
}
});
}));
// Now we have imagePromises, an array of Promises. When you have an array of Promises, you can use Promise.all to wait until all of them are resolved:
Promise.all(imagesPromises).then(responses => {
// Take a look at responses: it has to be an array of objects of two properties: dump and sample.
// Also, note that we finally use outer Promise's resolve function, resolveImagesCatalyst.
resolveImagesCatalyst(responses);
}).catch(errors => {
rejectImagesCatalyst(errors);
});
} else {
rejectImagesCatalyst('Sorry, nope.');
}
});
});
That's a huge one. What you really need to see is that
With Promise.all, you can wait for a collection of Promises to get resolved, and the "then" part won't get executed until all of them have been resolved.
You can put a Promise into a Promise, and resolve outer Promise when inner Promise gets resolved.
The code is really hard to read, because the order of execution is not top-to-bottom. If you use Webpack with Babel, you might want to take a look at async/await. With async/await, the code looks synchronous: you read it from top to bottom and that's exactly the order results of its execution appear, but under the hood, it's all asynchronous. Pretty neat ES6 feature, imho.

Make sure there is no existing Node module that is handles your imgur searching business. Search on npms.io.
If there is no existing module, find one that is close and expand it for your use case (hot images).
If you really can't find a module for imgur to expand, then make your own. All of the imgur request stuff goes in its own module (and own file).
Make sure that module supports promises.
Your code should look something like this:
import {getHotAlbums, getAlbumImages, config} from 'imgur';
config({clientID: 'BLAHXXXX'});
async function getHotImages() {
let hotAlbums = await getHotAlbums();
hotAlbums = hotAlbums.filter(a => a.imageCount > 50);
const sampleImages = [];
let albumIDs = hotAlbums.map(a => a.id);
for (let albumID of albumIDs) {
const images = await getAlbumImages(albumID);
const randomImageNum = Math.round(Math.random()*images.length)+1;
sampleImages.push(images[randomImageNum].link);
}
return sampleImages;
}