I'm trying to add a new property to an object. It seems that it works correctly from this scope:
rows.forEach(function (row, i) {
row.foo = i;
});
If I do console.log(rows) I can see that foo was added with the correct value. If I have another callback within the forEach, I don't see the change any more. Why?
rows.forEach(function (row, i) {
getUserById(row.user_id, function(user) {
row.foo = i;
});
});
Yes, the callback get's fired correctly. Here is the getUserById
function getUserById(userId, callback) {
connection.query('select * from t_user where id = ?', [userId], function(err, results) {
if (err) {
console.log("repo error");
} else {
if (results.length == 0) {
callback(null);
} else {
callback(results[0]);
}
}
});
}
I can only image that you would be seeing this issue if getUserById defers calling the callback function you pass it. On that assumption, your problem is a timing issue.
Each time you call getUserById and pass it a callback, you are basically saying "Call this function when you get the user for this id", and then going along with the rest of your program. That means that when console.log is called, your callbacks have not yet been called. Instead, you have to wait for all of your callbacks to finish, and then call any code that relies on the values you are setting in your callback.
The general approach to waiting for all your callbacks to finish is to craft a special callback function which will delegate to each callback and also keep track of once they've all been called.
Here is a question with an answer using the async library, which I'm a fan of (though there are a number of libraries to tackle this control flow problem).
Idiomatic way to wait for multiple callbacks in Node.js
Here's a raw JavaScript solution that is robust enough to work in the general case where not all the callbacks are the same (that is, if you wanted a different callback for each user id).
Multiple Asynchronous Callbacks - Using an array to fire function when complete
Related
I am trying to iterate over a list of keywords, and call mysql.query() for each one of them.
Problem is, I need each query call to end before the next one begins. How do I make that happen?
I have tried making it work with Promise requests but I do not wholly understand how asynchronous calls work.
keywords.forEach(keyword => {
sql.query("Select Id from Keyword Where Word= ?", [keyword],
function (err, ids) {...}
});
You can do it recursivly
function call(keywords, index){
if(keyworkds[index]){
sql.query("Select Id from Keyword Where Word= ?", [keyworkds[index].keyword],
function (err, ids) {
call(keywords, index + 1)
}
}
}
call(keywords, 0)
forEach will not wait for the asynchronous function to be done. The reason you need to pass a callback is because that's the only real way you have to do something after completion.
For that reason forEach simply will not work. It's possible to rewrite this to have the second iteration happen asynchronously, but it's ugly as hell.
I would recommend (if you can) to switch to async / await and use a mysql client that supports promises. Example:
for (const keyword of keywords) {
const result = await sql.query(
"SELECT Id from Keyword Where Word = ?",
[keyword]
);
}
You can use mysql2/promise from the mysql2 package to use promisified mysql function.
You won't be able to do this with just the forEach. Each call of sql.query will fire as quickly as the loop iterates and will not wait until it finishes. Also, there is no guarantee those queries will resolve in the order you call them in.
At the bare minimum, you can use callback functions, but that would be some really ugly and difficult code to deal with. https://en.wikipedia.org/wiki/Pyramid_of_doom_(programming)
That leaves Promises and async/await. I strongly recommend spending some time on this topic as you will be running into it a lot
https://javascript.info/async
Instead of looking for an async / loop solution, you can solve your original problem with one SQL query:
const args = keywords.map(_ => "?").join();
sql.query("Select Id, Word from Keyword Where Word in (" + args + ")", keywords,
function (err, records) {...}
);
I think async js library (https://caolan.github.io/async) is a good choice for something similar to your problem and with this library you have a cleaner code without nested async calls that produce Pyramid of doom. Whenever you face a problem that has many async calls that will run either in parallel or synchronously you can use it.
as the documentation said you can run only a single async operation at a time with series method like eachSeries.
async.eachSeries(keywords, function(keyword, callback) {
sql.query("Select Id from Keyword Where Word= ?", [keyword],
function (err, ids) {
if (err){
//if you want to stop reminding queries call callback with err
callback(err)
} else {
callback();
}
}
}, function(err) {
// if any of the queris produced an error, err would equal that error
if( err ) {
// One of the iterations produced an error.
// All processing will now stop.
console.log('A query failed to process');
}else {
console.log('All queries have been processed successfully');
}
});
forEach doesn't do anything special with async functions. It will call an async function, but it won't wait for it to complete before moving on. Therefore, if you need to call async promises sequentially, then you need another approach.
It looks like though you aren't using a library that returns a promise, so you will need to wrap the call in a promise (util.promisefy can help with that, or you can do it manually), or you can use a version of the library that does return promises. It is possible to do this with callbacks, but much harder.
The most straight forward is to use a for … of loop in side of an async function. In order to do this, you need a new enough version of javascript, and async functions are rather new. The basic idea look like this:
async function processCollection(collection) {
for (let item of collection) {
await processItem(item)
}
}
Where processItem would be another async function or a function that returns a promise (which are basically the same thing, just different syntaxes).
Another way, that doesn't require async functions, is to chain together promises. This can be done in a functional way as follows:
collection.reduce((previousPromise, item) => {
return previousPromose.then(() => processItem(item))
}, Promise.resolve(null))
(again, processItem is a function that returns a promise)
Whats goin on here is that by using reduce, you are basically calling promise.then(…).then(…).then(…)… once for each item in the collection. if the callback to then returns a promise, then it will wait for that promise to finish before calling the next then callback. So you get sequential execution. Also, any promise rejections are propagated as well, so errors will stop the next call from happening.
What is the best way to create parallel asynchronous HTTP requests and take the first result that comes back positive? I am familiar with the async library for JavaScript and would happy to use that but am not sure if it has exactly what I want.
Background - I have a Redis store that serves as state for a server. There is an API we can call to get some data that takes much longer than reaching the Redis store.
In most cases the data will already be in the Redis store, but in some cases it won't be there yet and we need to retrieve it from the API.
The simple thing to do would be to query Redis, and if the value is not in Redis then go to the API afterwards. However, we'll needlessly lose 20-50ms if the data is not yet in our Redis cache and we have to go to the API after failing to find the data with Redis. Since this particular API server is not under great load, it won't really hurt to go to the API simultaneously/in parallel, even if we don't absolutely need the returned value.
//pseudocode below
async.minimum([
function apiRequest(cb){
request(opts,function(err,response,body){
cb(err,body.result.hit);
}
},
function redisRequest(cb){
client.get("some_key", function(err, reply) {
cb(err,reply.result.hit);
});
}],
function minimumCompleted(err,result){
// this mimimumCompleted final callback function will be only fired once,
// and would be fired by one of the above functions -
// whichever one *first* returned a defined value for result.hit
});
is there a way to get what I am looking for with the async library or perhaps promises, or should I implement something myself?
Use Promise.any([ap, bp]).
The following is a possible way to do it without promises. It is untested but should meet the requirements.
To meet requirement of returning the first success and not just the first completion, I keep a count of the number of completions expected so that if an error occurs it can be ignored it unless it is the last error.
function asyncMinimum(a, cb) {
var triggered = false;
var completions = a.length;
function callback(err, data) {
completions--;
if (err && completions !== 0) return;
if (triggered) return;
triggered = true;
return cb(err, data);
}
a.map(function (f) { return f(callback); });
}
asyncMinimum([
function apiRequest(cb){
request(opts,function(err,response,body){
cb(err,body.result.hit);
}
},
function redisRequest(cb){
client.get("some_key", function(err, reply) {
cb(err,reply.result.hit);
});
}],
function minimumCompleted(err,result){
// this mimimumCompleted final callback function will be only fired once,
// and would be fired by one of the above functions -
// whichever one had a value for body.result.hit that was defined
});
The async.js library (and even promises) keep track of the number of asynchronous operations pending by using a counter. You can see a simple implementation of the idea in an answer to this related question: Coordinating parallel execution in node.js
We can use the same concept to implement the minimum function you want. Only, instead of waiting for the counter to count all responses before triggering a final callback, we deliberately trigger the final callback on the first response and ignore all other responses:
// IMHO, "first" is a better name than "minimum":
function first (async_functions, callback) {
var called_back = false;
var cb = function () {
if (!called_back) {
called_back = true; // block all other responses
callback.apply(null,arguments)
}
}
for (var i=0;i<async_functions.length;i++) {
async_functions[i](cb);
}
}
Using it would be as simple as:
first([apiRequest,redisRequest],function(err,result){
// ...
});
Here's an approach using promises. It takes a little extra custom code because of the non-standard result you're looking for. You aren't just looking for the first one to not return an error, but you're looking for the first one that has a specific type of result so that takes a custom result checker function. And, if none get a result, then we need to communicate that back to the caller by rejecting the promise too. Here's the code:
function firstHit() {
return new Promise(function(resolve, reject) {
var missCntr = 0, missQty = 2;
function checkResult(err, val) {
if (err || !val) {
// see if all requests failed
++missCntr;
if (missCntr === missQty) {
reject();
}
} else {
resolve(val);
}
}
request(opts,function(err, response, body){
checkResult(err, body.result.hit);
}
client.get("some_key", function(err, reply) {
checkResult(err, reply.result.hit);
});
});
}
firstHit().then(function(hit) {
// one of them succeeded here
}, function() {
// neither succeeded here
});
The first promise to call resolve() will trigger the .then() handler. If both fail to get a hit, then it will reject the promise.
I am struggling with getting my head around how to overcome and handle the async nature of Node.JS. I have done quite a bit of reading on it and tried to make Node do what I want by either using a message passing solution or callback functions.
My problem is I have a object where I want to constructor to load a file and populate an array. Then I want all calls to this function use that loaded data. So I need the original call to wait for the file to be loaded and all subsequent calls to use the already loaded private member.
My issue is that the function to load load the data and get the data is being executed async even if it return a function with a callback.
Anyways, is there something simple I am missing? Or is there an easier pattern I could use here? This function should return part of the loaded file but returns undefined. I have checked that the file is actually being loaded, and works correctly.
function Song() {
this.verses = undefined;
this.loadVerses = function(verseNum, callback) {
if (this.verses === undefined) {
var fs = require('fs'),
filename = 'README.md';
fs.readFile(filename, 'utf8', function(err, data) {
if (err) {
console.log('error throw opening file: %s, err: %s', filename, err);
throw err;
} else {
this.verses = data;
return callback(verseNum);
}
});
} else {
return callback(verseNum);
}
}
this.getVerse = function(verseNum) {
return this.verses[verseNum + 1];
}
}
Song.prototype = {
verse: function(input) {
return this.loadVerses(input, this.getVerse);
}
}
module.exports = new Song();
Update:
This is how I am using the song module from another module
var song = require('./song');
return song.verse(1);
"My issue is that the function to load the data and get the data is being executed async even if it return a function with a callback."
#AlbertoZaccagni what I mean by that scentence is that this line return this.loadVerses(input, this.getVerse); returns before the file is loaded when I expect it to wait for the callback.
That is how node works, I will try to clarify it with an example.
function readFile(path, callback) {
console.log('about to read...');
fs.readFile(path, 'utf8', function(err, data) {
callback();
});
}
console.log('start');
readFile('/path/to/the/file', function() {
console.log('...read!');
});
console.log('end');
You are reading a file and in the console you will likely have
start
about to read...
end
...read!
You can try that separately to see it in action and tweak it to understand the point. What's important to notice here is that your code will keep on running skipping the execution of the callback, until the file is read.
Just because you declared a callback does not mean that the execution will halt until the callback is called and then resumed.
So this is how I would change that code:
function Song() {
this.verses = undefined;
this.loadVerses = function(verseNum, callback) {
if (this.verses === undefined) {
var fs = require('fs'),
filename = 'README.md';
fs.readFile(filename, 'utf8', function(err, data) {
if (err) {
console.log('error throw opening file: %s, err: %s', filename, err);
throw err;
} else {
this.verses = data;
return callback(verseNum);
}
});
} else {
return callback(verseNum);
}
}
}
Song.prototype = {
verse: function(input, callback) {
// I've removed returns here
// I think they were confusing you, feel free to add them back in
// but they are not actually returning your value, which is instead an
// argument of the callback function
this.loadVerses(input, function(verseNum) {
callback(this.verses[verseNum + 1]);
});
}
}
module.exports = new Song();
To use it:
var song = require('./song');
song.verse(1, function(verse) {
console.log(verse);
});
I've ignored
the fact that we're not treating the error as first argument of the callback
the fact that calling this fast enough will create racing conditions, but I believe this is another question
[Collected into an answer and expanded from my previous comments]
TL;DR You need to structure your code such that the result of any operation is only used inside that operation's callback, since you do not have access to it anywhere else.
And while assigning it to an external global variable will certainly work as expected, do so will only occur after the callback has fired, which happens at a time you cannot predict.
Commentary
Callbacks do not return values because by their very nature, they are executed sometime in the future.
Once you pass a callback function into a controlling asynchronous function, it will be executed when the surrounding function decides to call it. You do not control this, and so waiting for a returned result won't work.
Your example code, song.verse(1); cannot be expected to return anything useful because it is called immediately and since the callback hasn't yet fired, will simply return the only value it can: null.
I'm afraid this reliance on asynchronous functions with passed callbacks is an irremovable feature of how NodeJS operates; it is at the very core of it.
Don't be disheartened though. A quick survey of all the NodeJS questions here shows quite clearly that this idea that one must work with the results of async operations only in their callbacks is the single greatest impediment to anyone understanding how to program in NodeJS.
For a truly excellent explanation/tutorial on the various ways to correctly structure NodeJS code, see Managing Node.js Callback Hell with Promises, Generators and Other Approaches.
I believe it clearly and succinctly describes the problem you face and provides several ways to refactor your code correctly.
Two of the features mentioned there, Promises and Generators, are programming features/concepts, the understanding of which would I believe be of great use to you.
Promises (or as some call them, Futures) is/are a programming abstraction that allows one to write code a little more linearly in a if this then that style, like
fs.readFileAsync(path).then(function(data){
/* do something with data here */
return result;
}).catch(function(err){
/* deal with errors from readFileAsync here */
}).then(function(result_of_last_operation){
/* do something with result_of_last_operation here */
if(there_is_a_problem) throw new Error('there is a problem');
return final_result;
})
.catch(function(err){
/* deal with errors when there_is_a_problem here */
}).done(function(final_result){
/* do something with the final result */
});
In reality, Promises are simply a means of marshaling the standard callback pyramid in a more linear fashion. (Personally I believe they need a new name, since the idea of "a promise of some value that might appear in the future" is not an easy one to wrap one's head around, at least it wasn't for me.)
Promises do this by (behind the scenes) restructuring "callback hell" such that:
asyncFunc(args,function callback(err,result){
if(err) throw err;
/* do something with the result here*/
});
becomes something more akin to:
var p=function(){
return new Promise(function(resolve,reject){
asyncFunc(args,function callback(err,result){
if(err) reject(err)
resolve(result);
});
});
});
p();
where any value you provide to resolve() becomes the only argument to the next "then-able" callback and any error is passed via rejected(), so it can be caught by any .catch(function(err){ ... }) handlers you define.
Promises also do all the things you'd expect from the (somewhat standard) async module, like running callbacks in series or in parallel and operating over the elements of an array, returning their collected results to a callback once all the results have been gathered.
But you will note that Promises don't quite do what you want, because everything is still in callbacks.
(See bluebird for what I believe is the simplest and thus, best Promises package to learn first.)
(And note that fs.readFileAsync is not a typo. One useful feature of bluebird is that it can be made to add this and other Promises-based versions of fs's existing functions to the standard fs object. It also understands how to "promisify" other modules such as request and mkdirp).
Generators are the other feature described in the tutorial above, but are only available in the new, updated but not yet officially released version of JavaScript (codenamed "Harmony").
Using generators would also allow you to write code in a more linear manner, since one of the features it provides is the ability of waiting on the results of an asynchronous operation in a way that doesn't wreak havoc with the JavaScript event-loop. (But as I said, it's not a feature in general use yet.)
You can however use generators in the current release of node if you'd like, simply add "--harmony" to the node command line to tell it to turn on the newest features of the next version of JavaScript.
I was unsure how node.js was able to realize what functions where async and which were not and how to create a custom async function.
Say I wanted to create a custom asynchronous function. I would be surprised if just because I called my last argument to the async function callback or cb that it would just know its an async function:
function f(arg1, callback){
//do stuff with arg1
arg1.doStuff()
//call callback
callback(null, arg1.result());
}
I tried something like that and it did not work async. How do you tell node.js that f is actually async?
NOTE: this answer was written in 2014, before the existence of async function, and before Promises gaining popularity. While the same principles apply as well, I would recommend reading on Promises before trying to get your head around how they relate to "traditional" callback-driven async functions.
To create a function that calls its callback asynchronously, you have to use some platform-provided async primitive (typically IO-related) on it - timers, reading from the filesystem, making a request etc.
For example, this function takes a callback argument, and calls it 100ms after:
function asyncFn(callback) {
setTimeout(() => {
callback();
}, 100);
}
A possible reason for making a function async when it doesn't need to be, is for API consistency. For example, suppose you have a function that makes a network request, and caches the result for later calls:
var cache = null;
function makeRequest(callback) {
if (!cache) {
makeAjax(result => {
cache = result;
callback(result);
});
} else {
callback(cache);
}
}
The problem is, this function is inconsistent: sometimes it is asynchronous, sometimes it isn't. Suppose you have a consumer like this:
makeRequest(result => doSomethingWithResult(result));
doSomethingElse();
The doSomethingElse function may run before or after the doSomethingWithResult function, depending on whether the result was cached or not. Now, if you use an async primitive on the makeRequest function, such as process.nextTick:
var cache = null;
function makeRequest(callback) {
if(!cache) {
makeAjax(result => {
cache = result;
callback(result);
});
} else {
process.nextTick(() => callback(cache));
}
}
The call is always async, and doSomethingElse always runs before doSomethingWithResult.
Only native functions (with access to the event loop) are asynchronous. You would need to call one of them to get asynchronity for your callback. See What is a simple example of an asynchronous javascript function?.
If you aren't using any, there's hardly a reason to make your function asynchronous.
I am quite new (just started this week) to Node.js and there is a fundamental piece that I am having trouble understanding. I have a helper function which makes a MySQL database call to get a bit of information. I then use a callback function to get that data back to the caller which works fine but when I want to use that data outside of that callback I run into trouble. Here is the code:
/** Helper Function **/
function getCompanyId(token, callback) {
var query = db.query('SELECT * FROM companies WHERE token = ?', token, function(err, result) {
var count = Object.keys(result).length;
if(count == 0) {
return;
} else {
callback(null, result[0].api_id);
}
});
}
/*** Function which uses the data from the helper function ***/
api.post('/alert', function(request, response) {
var data = JSON.parse(request.body.data);
var token = data.token;
getCompanyId(token, function(err, result) {
// this works
console.log(result);
});
// the problem is that I need result here so that I can use it else where in this function.
});
As you can see I have access to the return value from getCompanyId() so long as I stay within the scope of the callback but I need to use that value outside of the callback. I was able to get around this in another function by just sticking all the logic inside of that callback but that will not work in this case. Any insight on how to better structure this would be most appreciated. I am really enjoying Node.js thus far but obviously I have a lot of learning to do.
Short answer - you can't do that without violating the asynchronous nature of Node.js.
Think about the consequences of trying to access result outside of your callback - if you need to use that value, and the callback hasn't run yet, what will you do? You can't sleep and wait for the value to be set - that is incompatible with Node's single threaded, event-driven design. Your entire program would have to stop executing whilst waiting for the callback to run.
Any code that depends on result should be inside the getCompanyId callback:
api.post('/alert', function(request, response) {
var data = JSON.parse(request.body.data);
var token = data.token;
getCompanyId(token, function(err, result) {
//Any logic that depends on result has to be nested in here
});
});
One of the hardest parts about learning Node.js (and async programming is general) is learning to think asynchronously. It can be difficult at first but it is worth persisting. You can try to fight and code procedurally, but it will inevitably result in unmaintainable, convoluted code.
If you don't like the idea of multiple nested callbacks, you can look into promises, which let you chain methods together instead of nesting them. This article is a good introduction to Q, one implementation of promises.
If you are concerned about having everything crammed inside the callback function, you can always name the function, move it out, and then pass the function as the callback:
getCompanyId(token, doSomethingAfter); // Pass the function in
function doSomethingAfter(err, result) {
// Code here
}
My "aha" moment came when I began thinking of these as "fire and forget" methods. Don't look for return values coming back from the methods, because they don't come back. The calling code should move on, or just end. Yes, it feels weird.
As #joews says, you have to put everything depending on that value inside the callback(s).
This often requires you passing down an extra parameter(s). For example, if you are doing a typical HTTP request/response, plan on sending the response down every step along the callback chain. The final callback will (hopefully) set data in the response, or set an error code, and then send it back to the user.
If you want to avoid callback smells you need to use Node's Event Emitter Class like so:
at top of file require event module -
var emitter = require('events').EventEmitter();
then in your callback:
api.post('/alert', function(request, response) {
var data = JSON.parse(request.body.data);
var token = data.token;
getCompanyId(token, function(err, result) {
// this works
console.log(result);
emitter.emit('company:id:returned', result);
});
// the problem is that I need result here so that I can use it else where in this function.
});
then after your function you can use the on method anywhere like so:
getCompanyId(token, function(err, result) {
// this works
console.log(result);
emitter.emit('company:id:returned', result);
});
// the problem is that I need result here so that I can use it else where in this function.
emitter.on('company:id:returned', function(results) {
// do what you need with results
});
just be careful to set up good namespacing conventions for your events so you don't get a mess of on events and also you should watch the number of listeners you attach, here is a good link for reference:
http://www.sitepoint.com/nodejs-events-and-eventemitter/