This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 5 years ago.
Using NodeJS, I'm trying to get info from the database on a list of IDs and populate an array of objects. I need this to process synchronously. I'm having trouble figuring out how to do this so the next function call will wait for the previous one to finish before being called. Then all of the 'each' iterations to finish for the return to be called.
Example:
getInfo();
function getInfo(){
var return_arr = Array();
var ids = getListOfIDs();
// loop through each id getting more info from db
$.each( ids, function( i, v ){
return_arr.push(getFullInfo( id ));
});
return return_arr;
}
function getListOfIDs(){
// database call to get IDs
// returns array
}
function getFullInfo(){
// database call to get full info
// returns object
}
This is a simplified example, so assume that a single query to get all info will not work as there are multiple joins going on and post processing that needs to take place in js. Also, assume I'm doing proper error handling, which I omitted in my example.
Your database queries are asynchronous so you need to either use callbacks or promises to perform your work once the database returns with results.
function getListOfIDs(callback) {
// database call to get IDs
db.query('...', function(data) {
// call the callback and pass it the array
callback(data);
});
}
function getInfo() {
// pass an anonymous function as a callback
getListofIDs(function(data) {
// we now have access to the data passed into the callback
console.log(data);
});
}
Your current code example is synchronous, although I don't know how your db code works. The each loop is synchronous it just iterates over your ids and calls the getFullInfo function.
I'm not sure why you want synchronous code though as it doesn't utilise Nodes event loop architecture.
What I would do is use a good Promises framework such as Bluebird (http://bluebirdjs.com/docs/api/promise.each.html) with Promise.each ensuring each iteration occurs serially. Or you could also use a Callback library such as async (http://caolan.github.io/async/) with async.eachSeries. Either of these will ensure that you (a) get the benefit of asynchronous and (b) iterate in a serial fashion.
Promise way to do it:
function getListOfIDs() {
return new Promise((resolve, reject) => {
// database call
if (err)
{
reject(your_db_err);
return;
}
resolve(your_db_result);
});
}
function getInfo() {
getListOfIDs().then((result) => {
//do whatever you want with result
})
.catch((err) => {
//handle your err
});
}
Related
This question already has answers here:
How do I convert an existing callback API to promises?
(24 answers)
Closed 2 years ago.
Supposing that I have a Post schema, I want a method with this structure here:
function getPostDetails(post_id)
{
// The function searches the db for a document with that id,
// Finds it, and returns an object containing some of its details:
return {
title: post.title,
thumbnail: post.thumbnail,
views: post.views
// ....
};
}
It must have this form, I have a lot of async, lambda, and nested functions and I don't want to make that code messier...
I did a lot of research but didn't find the way of doing it, maybe because I suck at handling promises and async code? yes!
Mongoose calls are done asynchronously. They cannot be made synchronously and it is also bad idea to make DB calls synchronous.
You have 2 options, either you make function asynchronous to return promise or add a callback parameter.
Using async await.
async function getPostDetails(post_id) {
const post = await queryDb(post_id)
const data = map(post) // convert generic post to desire schema
return data
}
Using promises without async / await.
function getPostDetails(post_id) {
return queryDb(post_id).then(post => map(post))
}
Using callback.
function getPostDetails(post_id, callback) {
queryDb(post_id).then(post => map(post)).then(post => callback(post))
}
What is the best way to create parallel asynchronous HTTP requests and take the first result that comes back positive? I am familiar with the async library for JavaScript and would happy to use that but am not sure if it has exactly what I want.
Background - I have a Redis store that serves as state for a server. There is an API we can call to get some data that takes much longer than reaching the Redis store.
In most cases the data will already be in the Redis store, but in some cases it won't be there yet and we need to retrieve it from the API.
The simple thing to do would be to query Redis, and if the value is not in Redis then go to the API afterwards. However, we'll needlessly lose 20-50ms if the data is not yet in our Redis cache and we have to go to the API after failing to find the data with Redis. Since this particular API server is not under great load, it won't really hurt to go to the API simultaneously/in parallel, even if we don't absolutely need the returned value.
//pseudocode below
async.minimum([
function apiRequest(cb){
request(opts,function(err,response,body){
cb(err,body.result.hit);
}
},
function redisRequest(cb){
client.get("some_key", function(err, reply) {
cb(err,reply.result.hit);
});
}],
function minimumCompleted(err,result){
// this mimimumCompleted final callback function will be only fired once,
// and would be fired by one of the above functions -
// whichever one *first* returned a defined value for result.hit
});
is there a way to get what I am looking for with the async library or perhaps promises, or should I implement something myself?
Use Promise.any([ap, bp]).
The following is a possible way to do it without promises. It is untested but should meet the requirements.
To meet requirement of returning the first success and not just the first completion, I keep a count of the number of completions expected so that if an error occurs it can be ignored it unless it is the last error.
function asyncMinimum(a, cb) {
var triggered = false;
var completions = a.length;
function callback(err, data) {
completions--;
if (err && completions !== 0) return;
if (triggered) return;
triggered = true;
return cb(err, data);
}
a.map(function (f) { return f(callback); });
}
asyncMinimum([
function apiRequest(cb){
request(opts,function(err,response,body){
cb(err,body.result.hit);
}
},
function redisRequest(cb){
client.get("some_key", function(err, reply) {
cb(err,reply.result.hit);
});
}],
function minimumCompleted(err,result){
// this mimimumCompleted final callback function will be only fired once,
// and would be fired by one of the above functions -
// whichever one had a value for body.result.hit that was defined
});
The async.js library (and even promises) keep track of the number of asynchronous operations pending by using a counter. You can see a simple implementation of the idea in an answer to this related question: Coordinating parallel execution in node.js
We can use the same concept to implement the minimum function you want. Only, instead of waiting for the counter to count all responses before triggering a final callback, we deliberately trigger the final callback on the first response and ignore all other responses:
// IMHO, "first" is a better name than "minimum":
function first (async_functions, callback) {
var called_back = false;
var cb = function () {
if (!called_back) {
called_back = true; // block all other responses
callback.apply(null,arguments)
}
}
for (var i=0;i<async_functions.length;i++) {
async_functions[i](cb);
}
}
Using it would be as simple as:
first([apiRequest,redisRequest],function(err,result){
// ...
});
Here's an approach using promises. It takes a little extra custom code because of the non-standard result you're looking for. You aren't just looking for the first one to not return an error, but you're looking for the first one that has a specific type of result so that takes a custom result checker function. And, if none get a result, then we need to communicate that back to the caller by rejecting the promise too. Here's the code:
function firstHit() {
return new Promise(function(resolve, reject) {
var missCntr = 0, missQty = 2;
function checkResult(err, val) {
if (err || !val) {
// see if all requests failed
++missCntr;
if (missCntr === missQty) {
reject();
}
} else {
resolve(val);
}
}
request(opts,function(err, response, body){
checkResult(err, body.result.hit);
}
client.get("some_key", function(err, reply) {
checkResult(err, reply.result.hit);
});
});
}
firstHit().then(function(hit) {
// one of them succeeded here
}, function() {
// neither succeeded here
});
The first promise to call resolve() will trigger the .then() handler. If both fail to get a hit, then it will reject the promise.
I have a function which creates a database object out of three arrays. The arrays are filled in an each loop, one of the arrays relies on the value in the same iteration of the loop.
The dependent array uses the requests library and the cheerio library to grab a string to populate the array with.
Currently the dependent array fills with nulls which I think is because the loop is not waiting for the request to be returned.
I am still learning and would like to get this to work without direct blocking to keep things asynchronous so I'm looking into promises/callbacks.
This is being done server-side but from what I've seen in cheerios docs there is no promises capability.
Here's what I have so far. (getFile() is the function that isn't filling the 'c' array, it also depends on the current value being put into 'b'). I do know that the getFile function gets the correct value with a console log test, so the issue must be in the implementation of filling 'c'.
addToDB() is a function which saves a value into mongoDB, from testing I know that the objects are correctly being put into the db, just the c array is not correct.
function getInfo(path) {
$(path).each(function(i,e) {
a.push(...)
b.push(value)
c.push(getFile(value))
})
var entry = new DB...//(a,b,c)
addToDB(entry);
}
function getFile(...) {
request(fullUrl, function (err, resp, page) {
if (!err && resp.statusCode == 200) {
var $ = cheerio.load(page); // load the page
srcEp = $(this).attr("src");
return srcEp;
} // end error and status code
}); // end request
}
I've been reading about promises/callbacks and then() but I've yet to find anything which works.
First, you have to get your mind around the fact that any process that relies, at least in part, on an asynchronous sub-process, is itself inherently asynchronous.
At the lowest level of this question's code, request() is asynchronous, therefore its caller, getFile() is asynchronous, and its caller, getInfo() is also asynchronous.
Promises are an abstraction of the outcome of asynchronous processes and help enormously in coding the actions to be taken when such processes complete - successfully or under failure.
Typically, low-level asynchronous functions should return a promise to be acted on by their callers, which will, in turn, return a promise to their callers, and so on up the call stack. Inside each function, returned promise(s) may be acted on using promise methods, chiefly .then(), and may be aggregated using Promise.all() for example (syntax varies).
In this question, there is no evidence that request() currently returns a promise. You have three options :
discover that request() does, in fact, return a promise.
rewrite request() to return a promise.
write an adapter function (a "promisifier") that calls request(), and generates/returns the promise, which is later fulfilled or rejected depending on the outcome of request().
The first or second options would be ideal but the safe assumption for me (Roamer) is to assume that an adapter is required. Fortunately, I know enough from the question to be able to write one. Cheerio appears not to include jQuery's promise implementation, so a dedicated promise lib will be required.
Here is an adapter function, using syntax that will work with the Bluebird lib or native js promises:
//Promisifier for the low level function request()
function requestAsync(url) {
return new Promise(function(resolve, reject) {
request(url, function(err, resp, page) {
if (err) {
reject(err);
} else {
if (resp.statusCode !== 200) {
reject(new Error('request error: ' + resp.statusCode));
}
} else {
resolve(page);
}
});
});
}
Now getFile(...) and getInfo() can be written to make use of the promises returned from the lowest level's adapter.
//intermediate level function
function getFile(DOMelement) {
var fullUrl = ...;//something derived from DOMelement. Presumably either .val() or .text()
return requestAsync(fullUrl).then(function (page) {
var $ = cheerio.load(page);
srcEp = $(???).attr('src');//Not too sure what the selector should be. `$(this)` definitely won't work.
return srcEp;
});
}
//high level function
function getInfo(path) {
var a = [], b = [], c = [];//presumably
// Now map the $(path) to an array of promises by calling getFile() inside a .map() callback.
// By chaining .then() a/b/c are populated when the async data arrives.
var promises = $(path).map(function(i, e) {
return getFile(e).then(function(srcEp) {
a[i] = ...;
b[i] = e;
c[i] = srcEp;
});
});
//return an aggregated promise to getInfo's caller,
//in case it needs to take any action on settlement.
return Promise.all(promises).then(function() {
//What to do when all promises are fulfilled
var entry = new DB...//(a,b,c)
addToDB(entry);
}, function(error) {
//What to do if any of the promises fails
console.log(error);
//... you may want to do more.
});
}
This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 7 years ago.
I am trying to use async to collect some additional data to my array.
for this purpose ive created the following:
User_has_academy_module.findAll({
include: [{model: Module, include: [Category, ModuleType]}],
where: {academy_team_id: team_id, user_id: user_id}
}).then(function (modules) {
var i = 0;
async.map(modules, function (module,moduleCallback) {
var act = Academy_Attempt.build();
if(module.dataValues.is_complete == 1){
act.findByUserTeamAndModule(module.dataValues.user_id, module.dataValues.academy_team_id, module.dataValues.module_id, function (result) {
module.dataValues.academy_attempt = result;
moduleCallback(null, module);
}, function (error) {
})
}
});
onSuccess(modules);
})
as you can see from the above i first collect an array called modules that i need loop over for each of these modules i want to find additonal data if a value called is_complete == 1
once it finds a value it should set the module.dataValues.academy_attempt = result
Once all of the modules have been iterated it should call the callback (onSuccess) and return the modules.
However it runs onSuccess before the async call.
So my question is what am i doing wrong and how can fix it?
You need to run onSuccess in a completion callback to async.map. The first callback is the iterator, runs for every mapped element; the second callback (which you currently aren't using) runs after all iterations have completed.
async.map(modules, function (module,moduleCallback) {
//...
}, function(err, modules) {
onSuccess(modules);
});
Currently, your code queues up the asynchronous map job and then runs onSuccess without waiting for map to finish (or even start).
This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 8 years ago.
I'm calling a C# api through Jquery AJAX. I am able to loop through all of the data and print each in the console; however, I am not able return the data to a variable (var x) so that it contains all objects returned from the api call.
This is my code thus far:
var uri = 'api/products';
function test(){
// Send an AJAX request
$.getJSON(uri)
.done(function (data) {
// On success, 'data' contains a list of products.
var info = {};
$.each(data, function (key, item) {
// Add a list item for the product.
console.log(item);
info.append(item);
});
return info;
});
}
var x = test();
I have tried to just simply skipping the $.each() and returning data, with no luck.
Should I be using a method other than append? Or is there a way to just simply return data?
Thanks.
.done() is a promise (http://api.jquery.com/promise/), which takes a callback function. The callback function is void; nothing consumes its return...and the return in your code is the return for the (anonymous) callback function, not for test(), which is why you're seeing no value come out.
This is an asynchronous pattern. That means test() is going to return immediately, not waiting for the AJAX call to complete, and there's no value yet to put into x! The AJAX response will arrive later, at some unknown time. The purpose of the callback function is to allow you to (at that time!) do whatever it is you intend to do with the value you receive. You haven't shown us what that is in this example, but you're going to need to restructure it to happen after the promise is fulfilled.